{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Crosses\n",
    "\n",
    " A feature cross is a synthetic feature formed by multiplying two or more features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.cloud import bigquery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Crosses in BigQuery\n",
    "\n",
    "We'll first explore how to create a feature cross in BigQuery. The cell below will create a dataset called `babyweight` in your GCP project, if it does not already exist. This dataset will will house our tables and models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset already exists.\n"
     ]
    }
   ],
   "source": [
    "bq = bigquery.Client()\n",
    "dataset = bigquery.Dataset(bq.dataset(\"babyweight\"))\n",
    "\n",
    "try:\n",
    "    bq.create_dataset(dataset)\n",
    "    print(\"Dataset created.\")\n",
    "except:\n",
    "    print(\"Dataset already exists.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create datasets for training and evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "CREATE OR REPLACE TABLE\n",
    "    babyweight.babyweight_data AS\n",
    "SELECT\n",
    "    weight_pounds,\n",
    "    CAST(is_male AS STRING) AS is_male,\n",
    "    mother_age,\n",
    "    CASE\n",
    "        WHEN plurality = 1 THEN \"Single(1)\"\n",
    "        WHEN plurality = 2 THEN \"Twins(2)\"\n",
    "        WHEN plurality = 3 THEN \"Triplets(3)\"\n",
    "        WHEN plurality = 4 THEN \"Quadruplets(4)\"\n",
    "        WHEN plurality = 5 THEN \"Quintuplets(5)\"\n",
    "    END AS plurality,\n",
    "    gestation_weeks,\n",
    "    CAST(mother_race AS STRING) AS mother_race,\n",
    "    FARM_FINGERPRINT(\n",
    "        CONCAT(\n",
    "            CAST(year AS STRING),\n",
    "            CAST(month AS STRING)\n",
    "        )\n",
    "    ) AS hashmonth\n",
    "FROM\n",
    "    publicdata.samples.natality\n",
    "WHERE\n",
    "    year > 2000\n",
    "    AND weight_pounds > 0\n",
    "    AND mother_age > 0\n",
    "    AND plurality > 0\n",
    "    AND gestation_weeks > 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll create tables in BigQuery that we'll use for training and evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "CREATE OR REPLACE TABLE\n",
    "    babyweight.babyweight_data_train AS\n",
    "SELECT\n",
    "    weight_pounds,\n",
    "    is_male,\n",
    "    mother_age,\n",
    "    plurality,\n",
    "    gestation_weeks,\n",
    "    mother_race\n",
    "FROM\n",
    "    babyweight.babyweight_data\n",
    "WHERE\n",
    "    ABS(MOD(hashmonth, 4)) < 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "CREATE OR REPLACE TABLE\n",
    "    babyweight.babyweight_data_eval AS\n",
    "SELECT\n",
    "    weight_pounds,\n",
    "    is_male,\n",
    "    mother_age,\n",
    "    plurality,\n",
    "    gestation_weeks,\n",
    "    mother_race\n",
    "FROM\n",
    "    babyweight.babyweight_data\n",
    "WHERE\n",
    "    ABS(MOD(hashmonth, 4)) = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create model in BigQuery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing query with job ID: 5863f92f-a405-466b-b226-2b940e1d13fb\n",
      "Query executing: 2028.03s"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bigquery\n",
    "CREATE OR REPLACE MODEL `babyweight.natality_model`\n",
    "OPTIONS\n",
    "  (MODEL_TYPE=\"DNN_REGRESSOR\",\n",
    "    HIDDEN_UNITS=[64, 32],\n",
    "    BATCH_SIZE=32,\n",
    "    INPUT_LABEL_COLS=[\"weight_pounds\"],\n",
    "    DATA_SPLIT_METHOD=\"NO_SPLIT\") AS\n",
    "SELECT\n",
    "  weight_pounds,\n",
    "  is_male,\n",
    "  plurality,\n",
    "  gestation_weeks,\n",
    "  mother_age,\n",
    "  CAST(mother_race AS string) AS mother_race\n",
    "FROM\n",
    "  babyweight.babyweight_data_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `ML.EVALUATE` to determine the root mean square error of our model on the evaluation set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"\"\"\n",
    "SELECT\n",
    "  *, SQRT(mean_squared_error) AS rmse\n",
    "FROM\n",
    "  ML.EVALUATE(MODEL `babyweight.natality_model`,\n",
    "    (\n",
    "    SELECT\n",
    "      weight_pounds,\n",
    "      is_male,\n",
    "      plurality,\n",
    "      gestation_weeks,\n",
    "      mother_age,\n",
    "      CAST(mother_race AS STRING) AS mother_race\n",
    "    FROM\n",
    "      babyweight.babyweight_data_eval ))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_absolute_error</th>\n",
       "      <th>mean_squared_error</th>\n",
       "      <th>mean_squared_log_error</th>\n",
       "      <th>median_absolute_error</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>explained_variance</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.827813</td>\n",
       "      <td>1.15396</td>\n",
       "      <td>0.023499</td>\n",
       "      <td>0.665818</td>\n",
       "      <td>0.336818</td>\n",
       "      <td>0.339768</td>\n",
       "      <td>1.074225</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_absolute_error  mean_squared_error  mean_squared_log_error  \\\n",
       "0             0.827813             1.15396                0.023499   \n",
       "\n",
       "   median_absolute_error  r2_score  explained_variance      rmse  \n",
       "0               0.665818  0.336818            0.339768  1.074225  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = bq.query(query).to_dataframe()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Feature Cross with BQML\n",
    "\n",
    "Next, we'll create a feature cross of the features `is_male` and `mother_race`. To create a feature cross we apply `ML.FEATURE_CROSS` to a STRUCT of the features `is_male` and `mother_race` cast as a string. \n",
    "The STRUCT clause creates an ordered pair of the two features. The TRANSFORM clause is used for engineering features of our model. This allows us to specify all preprocessing during model creation and apply those preprocessing steps during prediction and evaluation. The rest of the features within the TRANSFORM clause remain unchanged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing query with job ID: 9863811d-26df-40da-b957-1b3e03fe39f0\n",
      "Query executing: 5.11s"
     ]
    }
   ],
   "source": [
    "%%bigquery\n",
    "CREATE OR REPLACE MODEL `babyweight.natality_model_feat_eng`\n",
    "TRANSFORM(weight_pounds,\n",
    "    is_male,\n",
    "    plurality,\n",
    "    gestation_weeks,      \n",
    "    mother_age,\n",
    "    CAST(mother_race AS string) AS mother_race,\n",
    "    ML.FEATURE_CROSS(\n",
    "            STRUCT(\n",
    "                is_male,\n",
    "                plurality)\n",
    "        ) AS gender_X_plurality)\n",
    "OPTIONS\n",
    "  (MODEL_TYPE='linear_reg',\n",
    "   INPUT_LABEL_COLS=['weight_pounds'],\n",
    "   DATA_SPLIT_METHOD=\"NO_SPLIT\") AS    \n",
    "SELECT\n",
    "  *\n",
    "FROM\n",
    "    babyweight.babyweight_data_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we compute the root mean square error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"\"\"\n",
    "SELECT\n",
    "  *, SQRT(mean_squared_error) AS rmse\n",
    "FROM\n",
    "  ML.EVALUATE(MODEL `babyweight.natality_model_feat_eng`,\n",
    "    (\n",
    "    SELECT\n",
    "      weight_pounds,\n",
    "      is_male,\n",
    "      plurality,\n",
    "      gestation_weeks,\n",
    "      mother_age,\n",
    "      CAST(mother_race AS STRING) AS mother_race\n",
    "    FROM\n",
    "      babyweight.babyweight_data_eval ))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_absolute_error</th>\n",
       "      <th>mean_squared_error</th>\n",
       "      <th>mean_squared_log_error</th>\n",
       "      <th>median_absolute_error</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>explained_variance</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.819834</td>\n",
       "      <td>1.115964</td>\n",
       "      <td>0.020036</td>\n",
       "      <td>0.664964</td>\n",
       "      <td>0.358654</td>\n",
       "      <td>0.358656</td>\n",
       "      <td>1.056392</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_absolute_error  mean_squared_error  mean_squared_log_error  \\\n",
       "0             0.819834            1.115964                0.020036   \n",
       "\n",
       "   median_absolute_error  r2_score  explained_variance      rmse  \n",
       "0               0.664964  0.358654            0.358656  1.056392  "
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = bq.query(query).to_dataframe()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Crosses in Keras\n",
    "\n",
    "Next, we'll see how to implement a feature cross in Tensorflow using feature columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import tensorflow as tf\n",
    "import datetime\n",
    "\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow import feature_column as fc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determine CSV, label, and key columns\n",
    "# Create list of string column headers, make sure order matches.\n",
    "CSV_COLUMNS = [\"weight_pounds\",\n",
    "               \"is_male\",\n",
    "               \"mother_age\",\n",
    "               \"plurality\",\n",
    "               \"gestation_weeks\",\n",
    "               \"mother_race\"]\n",
    "\n",
    "# Add string name for label column\n",
    "LABEL_COLUMN = \"weight_pounds\"\n",
    "\n",
    "# Set default values for each CSV column as a list of lists.\n",
    "# Treat is_male and plurality as strings.\n",
    "DEFAULTS = [[0.0], [\"null\"], [0.0], [\"null\"], [0.0], [\"null\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a dataset of features and label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def features_and_labels(row_data):\n",
    "    \"\"\"Splits features and labels from feature dictionary.\n",
    "    Args:\n",
    "        row_data: Dictionary of CSV column names and tensor values.\n",
    "    Returns:\n",
    "        Dictionary of feature tensors and label tensor.\n",
    "    \"\"\"\n",
    "    label = row_data.pop(LABEL_COLUMN)\n",
    "\n",
    "    return row_data, label\n",
    "\n",
    "\n",
    "def load_dataset(pattern, batch_size=1, mode=tf.estimator.ModeKeys.EVAL):\n",
    "    \"\"\"Loads dataset using the tf.data API from CSV files.\n",
    "    Args:\n",
    "        pattern: str, file pattern to glob into list of files.\n",
    "        batch_size: int, the number of examples per batch.\n",
    "        mode: tf.estimator.ModeKeys to determine if training or evaluating.\n",
    "    Returns:\n",
    "        `Dataset` object.\n",
    "    \"\"\"\n",
    "    # Make a CSV dataset\n",
    "    dataset = tf.data.experimental.make_csv_dataset(\n",
    "        file_pattern=pattern,\n",
    "        batch_size=batch_size,\n",
    "        column_names=CSV_COLUMNS,\n",
    "        column_defaults=DEFAULTS)\n",
    "\n",
    "    # Map dataset to features and label\n",
    "    dataset = dataset.map(map_func=features_and_labels)  # features, label\n",
    "\n",
    "    # Shuffle and repeat for training\n",
    "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        dataset = dataset.shuffle(buffer_size=1000).repeat()\n",
    "\n",
    "    # Take advantage of multi-threading; 1=AUTOTUNE\n",
    "    dataset = dataset.prefetch(buffer_size=1)\n",
    "\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll need to get the data read in by our input function to our model function, but just how do we go about connecting the dots? We can use Keras input layers (tf.Keras.layers.Input)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_input_layers():\n",
    "    \"\"\"Creates dictionary of input layers for each feature.\n",
    "\n",
    "    Returns:\n",
    "        Dictionary of `tf.Keras.layers.Input` layers for each feature.\n",
    "    \"\"\"\n",
    "    inputs = {\n",
    "        colname: tf.keras.layers.Input(\n",
    "            name=colname, shape=(), dtype=\"float32\")\n",
    "        for colname in [\"mother_age\", \"gestation_weeks\"]}\n",
    "\n",
    "    inputs.update({\n",
    "        colname: tf.keras.layers.Input(\n",
    "            name=colname, shape=(), dtype=\"string\")\n",
    "        for colname in [\"is_male\", \"plurality\", \"mother_race\"]})\n",
    "\n",
    "    return inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create feature columns for inputs\n",
    "Next, define the feature columns. `mother_age` and `gestation_weeks` should be numeric. The others, `is_male`, `plurality` and `mother_race`, should be categorical. Remember, only dense feature columns can be inputs to a DNN.\n",
    "\n",
    "The last feature column created in the `create_feature_columns` function is a feature cross with `is_male` and `plurality`. To implement a feature cross in Tensorflow we use `tf.feature_column.crossed_column` which takes two arguments: a list of the feature keys to be crossed and the hash bucket size. Crossed features will be hashed according to `hash_bucket_size` so it should be large enough to accommodate all possible crossed categories. Since the feature `is_male` can take 3 values (True, False or Unknown) and the feature `plurality` can take 6 values (Single(1), Twins(2), Triplets(3), Quadruplets(4), Quintuplets(5), Multiple(2+)), we'll set `hash_bucket_size=18`.\n",
    "\n",
    "Finally, to use crossed column in DNN model, you need to wrap it either in an `indicator_column` or an `embedding_column`. In the code below, we use an embedding column and take the embedding dimension to be 2. \n",
    "\n",
    "To create a crossed column with features of numeric type, you can use `categorical_column`, or `bucketized_column` before passing to a `crossed_column`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def categorical_fc(name, values):\n",
    "    cat_column = fc.categorical_column_with_vocabulary_list(\n",
    "            key=name, vocabulary_list=values)\n",
    "\n",
    "    return fc.indicator_column(categorical_column=cat_column)\n",
    "\n",
    "\n",
    "def create_feature_columns():\n",
    "    feature_columns = {\n",
    "        colname : fc.numeric_column(key=colname)\n",
    "           for colname in [\"mother_age\", \"gestation_weeks\"]\n",
    "    }\n",
    "\n",
    "    feature_columns[\"is_male\"] = categorical_fc(\n",
    "        \"is_male\", [\"True\", \"False\", \"Unknown\"])\n",
    "    feature_columns[\"plurality\"] = categorical_fc(\n",
    "        \"plurality\", [\"Single(1)\", \"Twins(2)\", \"Triplets(3)\",\n",
    "                      \"Quadruplets(4)\", \"Quintuplets(5)\", \"Multiple(2+)\"])\n",
    "    feature_columns[\"mother_race\"] = fc.indicator_column(\n",
    "        fc.categorical_column_with_hash_bucket(\n",
    "            \"mother_race\", hash_bucket_size=17, dtype=tf.dtypes.string))\n",
    "    \n",
    "    feature_columns[\"gender_x_plurality\"] = fc.embedding_column(\n",
    "        fc.crossed_column([\"is_male\", \"plurality\"], hash_bucket_size=18),\n",
    "        dimension=2)\n",
    "\n",
    "    return feature_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can double-check the output of `create_feature_columns`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature column keys: \n",
      "['mother_age', 'gestation_weeks', 'is_male', 'plurality', 'mother_race', 'gender_x_plurality']\n",
      "\n",
      "Feature column values: \n",
      "[NumericColumn(key='mother_age', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='gestation_weeks', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='is_male', vocabulary_list=('True', 'False', 'Unknown'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='plurality', vocabulary_list=('Single(1)', 'Twins(2)', 'Triplets(3)', 'Quadruplets(4)', 'Quintuplets(5)', 'Multiple(2+)'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), IndicatorColumn(categorical_column=HashedCategoricalColumn(key='mother_race', hash_bucket_size=17, dtype=tf.string)), EmbeddingColumn(categorical_column=CrossedColumn(keys=('is_male', 'plurality'), hash_bucket_size=18, hash_key=None), dimension=2, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x7fcaf9611090>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, max_norm=None, trainable=True)]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "feature_columns = create_feature_columns()\n",
    "print(\"Feature column keys: \\n{}\\n\".format(list(feature_columns.keys())))\n",
    "print(\"Feature column values: \\n{}\\n\".format(list(feature_columns.values())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a DNN model\n",
    "Next we define our model. This is regression so make sure the output layer activation is correct and that the shape is right. We'll create deep neural network model, similar to what we use in BigQuery."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model_outputs(inputs):\n",
    "    # Create two hidden layers of [64, 32] just in like the BQML DNN\n",
    "    h1 = layers.Dense(64, activation=\"relu\", name=\"h1\")(inputs)\n",
    "    h2 = layers.Dense(32, activation=\"relu\", name=\"h2\")(h1)\n",
    "\n",
    "    # Final output is a linear activation because this is regression\n",
    "    output = layers.Dense(units=1, activation=\"linear\", name=\"weight\")(h2)\n",
    "\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rmse(y_true, y_pred):\n",
    "    return tf.sqrt(tf.reduce_mean((y_pred - y_true) ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will build the model using tf.keras.models.Model giving our inputs and outputs and then compile our model with an optimizer, a loss function, and evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here is our DNN architecture so far:\n",
      "\n",
      "Model: \"model_6\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "gestation_weeks (InputLayer)    [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "is_male (InputLayer)            [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "mother_age (InputLayer)         [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "mother_race (InputLayer)        [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "plurality (InputLayer)          [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense_features_6 (DenseFeatures (None, 30)           36          gestation_weeks[0][0]            \n",
      "                                                                 is_male[0][0]                    \n",
      "                                                                 mother_age[0][0]                 \n",
      "                                                                 mother_race[0][0]                \n",
      "                                                                 plurality[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "h1 (Dense)                      (None, 64)           1984        dense_features_6[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "h2 (Dense)                      (None, 32)           2080        h1[0][0]                         \n",
      "__________________________________________________________________________________________________\n",
      "weight (Dense)                  (None, 1)            33          h2[0][0]                         \n",
      "==================================================================================================\n",
      "Total params: 4,133\n",
      "Trainable params: 4,133\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "def build_dnn_model():\n",
    "    \"\"\"Builds simple DNN using Keras Functional API.\n",
    "\n",
    "    Returns:\n",
    "        `tf.keras.models.Model` object.\n",
    "    \"\"\"\n",
    "    # Create input layer\n",
    "    inputs = create_input_layers()\n",
    "\n",
    "    # Create feature columns\n",
    "    feature_columns = create_feature_columns()\n",
    "\n",
    "    # The constructor for DenseFeatures takes a list of numeric columns\n",
    "    # The Functional API in Keras requires: LayerConstructor()(inputs)\n",
    "    dnn_inputs = layers.DenseFeatures(\n",
    "        feature_columns=feature_columns.values())(inputs)\n",
    "\n",
    "    # Get output of model given inputs\n",
    "    output = get_model_outputs(dnn_inputs)\n",
    "\n",
    "    # Build model and compile it all together\n",
    "    model = tf.keras.models.Model(inputs=inputs, outputs=output)\n",
    "    model.compile(optimizer=\"adam\", loss=\"mse\", metrics=[rmse, \"mse\"])\n",
    "\n",
    "    return model\n",
    "\n",
    "print(\"Here is our DNN architecture so far:\\n\")\n",
    "model = build_dnn_model()\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/QAAAFhCAIAAABRc3bsAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdeVxTx/o/8AkJIouIsggUXFAEUQsubMJVlFUqLi24ImplUbEqtVZrq+LFCrRVsdYFUOqGVhbFvUoUqyBuCCJgXe/PK4gCKihEkOX8/pj7zeUCiSwhJ4TP+w9fyfFk5pk5k5yHkzkTDsMwBAAAAAAAOj4FtgMAAAAAAADJQHIPAAAAACAnkNwDAAAAAMgJJPcAAAAAAHKCV/9Jenr65s2b2QoFoHVsbW2//vprtqMAAAAAYN//XLl/9uxZQkICW6EAtMK1a9fS09PZjgIAAABAJvAab4qPj5d+HACt4+XlxXYIAAAAALICc+4BAAAAAOQEknsAAAAAADmB5B4AAAAAQE4guQcAAAAAkBNI7gEAAAAA5ASSewAAAAAAOYHkHgAAAABATiC5BwAAAACQE0juAQAAAADkBJJ7AAAAAAA5geQeAAAAAEBOILkHAAAAAJATSO4BAAAAAOQEknspYRgmKyurqqqK7UAAAAAAQG51vOQ+KSnJ0NDw3r17bAfSAocOHerfv/+wYcNKS0vbXhqfz/f19eVwOBwOx9XVNTY2tu1lihcfH29jY0NrXLp0aVZWVnvXCAAAAACtwHJyX1hY2NLdVFVVdXR0unbt2m5BSd7MmTM9PT0lVZqTk9Pu3bu1tbUJITExMbNmzZJUyQ0Iu93LyysiIoIQYmFhsXXrVgsLi3aqEQAAAADags3k/s2bN97e3i3dzdnZOSMjo1+/fu0ZmuRpampKtkB1dXVCSPfu3SVbrFCDbtfQ0GjX6gAAAACg7VhL7gUCwfTp0588eSKR3TohDocj/FfiGnd7u1YHAAAAABLRyuQ+IyPD399/5syZVlZWkZGRNTU1dDvDMLt27Vq4cKG1tbWLi8vDhw/p9qysrHnz5oWHh0+aNMnZ2ZkQcuzYsXv37pWUlPj5+f3yyy+EkJcvX/r5+YWEhPj5+U2ZMuXVq1eNd3vz5s2ePXucnZ2TkpKEwSQmJi5evPibb74ZP378Dz/8QG9azcrKWrFihZGRUUVFha+vr5aWlpWV1Uf/SDh37hyPx+vSpcupU6cqKyv9/Pw4HI6JicmlS5cIIf/+979tbGzoBBtRLRW1vb6TJ09yudxJkyYdO3asyc4hhKSlpRkaGp49e7Y5h0NMY/Py8r7//nszM7Pnz59Pnjy5Z8+eVlZW165dI4QcPnxYXV3d0NCQEFJWVhYSEsLlcm1tbZs8Oh/V5OE7fvx4t27dOBxORETEhw8fCCHp6el6enobN24U1VcFBQVhYWFDhgx5/fq1q6trnz59aFEAAAAA8HFMPUeOHGmwpUlPnz5VVVX917/+xTCMj48PIWTEiBHLli1jGCY0NHTv3r0Mw9TU1JiZmenq6lZUVDAMM3DgwNTUVIZhBAKBvb09LWfChAl9+/YVFuvg4DBt2jT62Nzc3Nvbu/FueXl5QUFBhJCEhAS6ZcuWLaNGjfrw4QPDMCUlJcbGxmPGjKmrqyssLHRyciKEBAYG5ubmZmZmKikpTZ8+/aOtmzFjRpcuXWjYHz58MDQ0dHJyEv6vl5fX48ePxbRU1PawsDBCyIsXLxiGWbVqVVRUlLDMJjvn9OnTysrKsbGxouIcMGAAIaS8vJxhGDGNXbVqlYaGBpfLDQoKSklJSUxM1NLSUlFRef78OcMwLi4uBgYGwjKHDh1qY2PT5NH5+++/CSEODg6i4hF1+FatWkUIuXnzJn1aVVVlbW1NHzfZV2fPnjU1NeVyuevWrYuKirKysiooKBBVKcMwnp6enp6eYnYAAAAA6Dxac+X+t99+69mzZ9++fQkhq1evJoT4+/tv2bLl+fPnERERs2fPJoRwuVxPT88XL16cPHmyurr64cOHGRkZhBBlZeXly5c3WSyHwzE3N6ePhwwZkp2d3XifQYMGTZo0Sfi0qKjohx9+WLBggaKiIiFEU1Nz9erVf/31V2xsrK6urqWlJSFk/fr1ZmZmFhYWlpaWNAbx/Pz8Pnz4QL8ZUFRU/OKLLy5fvvzmzRtCSGVlZW1trZGRkaiWitouLLyuru67774bNWqUn58f3SKqc9zd3d+9ezdz5syPBkwIEdPY0NBQd3d3BQWF8PBwBweHzz//fOfOnQKBYNeuXYQQFRWV+uWoqqo2p7omiTp8gYGBPB4vMjKSPk1OTp4wYQIhRFRfubm52dnZ1dbWent7+/n5Xb9+XV9fv9VRAQAAAHQqvFa8pqCgQCAQ0McmJiaamprPnj0jhFy9erW6ujogIEC4p6+vr7KysqKioqur67Jly3JycsLCwiZPntxksRcvXiSEVFZWxsbG3rhxg2GYpiPm/Tfma9euVVRU9O7dW7iFJo4pKSne3t5cLrf+/gYGBo8ePfpo6xwcHPr163fgwAGaWGdnZ9fU1MTHx/v7+ycmJn7xxRdiWipqu/BpYGCgrq6uh4eHcIuYzqHxN5OYxqqoqHC5XPr3DyFk8uTJSkpKd+/ebX7hzSHq8BkYGHh5eR08eDA0NFRLSysuLm7dunVEdB8SQhQVFXk8Hv1qAgAAAACarzXJvbu7+6FDhy5cuODo6FhaWlpRUeHm5kYIuXfvnqqqanR0dOOXJCYm+vn5RUdHHzt2LC4ubuzYsY33qa2t/emnn27durVkyRJra2s6L1y8p0+fEkJev34t3CKcc9KKdlEcDmfOnDkhISEvXrx49OiRlZUVl8s9ePAgTe7povKiWhoSEiKqBygVFZXo6OjZs2fTqe1UczpHgng8nr6+vvA2CUkRc/iCgoIOHz4cFRX1zTfflJSUGBkZEbGjBQAAAABapzXTcmbNmhUdHe3j47NmzZqvv/768OHDdnZ2hBAVFZX8/Pz8/Pz6OxcXFxNCeDxebGxsbGwsj8dzc3Nr/BNUdXV17u7ueXl5iYmJY8aMaWYkdEHMxrfJmpqatqJdQnPmzKmrqzt8+PD27du/+uqrOXPmpKamXrx4UU9Pj15aFtVSMT1A/fjjj6ampjNmzKj/a1Yf7RyJEwgEbeyi+h4+fCgQCMQcPktLSzs7u+3bt586dUr4rcVH+woAAAAAWqo1yT2dJn7nzp2QkJCYmBjhTJKhQ4cyDLNy5Urhno8fP96xY0dVVVVUVBQhZObMmdeuXWMYJiUlhRCioKBQXl5O97xx48b58+cdHByEVQjnddTfrQFbW1t1dfX6K+fk5+cLBIKJEye2ol1Cffv2dXBw2LZtm7Kysr6+/pQpU9TU1GbNmjVv3jzxLRW1Xfi0a9euBw4cKCwsFM65F9U5hJC6ujoxQdL+ETV5SYzCwsLi4mK65g+PxysvL6+traX/VV5eLqy0QbeLqohhmAULFmRmZoo6fNTy5cufP3++fPlyLy8vuuWjfQUAAAAALdWaaTnh4eF//fWXhYWFnp6empqapqYmvYLu7OxsaWl56NChysrKKVOmvH379ujRo3/88QchJCYmZuHChVwuV19fv3v37sOHDyeE6Ovrl5SUZGRkvHv3juaC+/bts7KyunnzZm5u7suXL7Ozs3v16lV/NysrK/qzqfQSr6amZnh4+KJFi+gcIULIr7/+OmfOHDqzpaysjBAinH9SVFQkvFXgo+bNm+fj43P06FFCiIqKipeX140bN0aOHEn/V1RLNTU1RfVARUUFDcbCwmL9+vXfffddaGjod999J6pz+Hz+F198sWfPHlE/bfv27VvaRjU1tY82tqqq6s6dO/SG1w0bNsyZM8fKyooQMnTo0ISEhNDQ0KlTp8bFxVVVVT179iwzM3PYsGENup2WX/8LB1rpkiVLevToQef6N3n4evXqRQiZOHFi7969zc3NhT/mJWa00L83SktL6S9nAQAAAEBz1V86p5lLYZ48ebJbt271Cxk8eDBdr/DVq1ezZs3S0dHR1tb28fGhGysrKy0tLV1dXcPCwvz9/aOjo2k5d+7cMTAwGDhwYHx8PL0A3K1bNxsbGz6ff+bMGS0tLU9Pz/Ly8vq7XbhwYfTo0YSQkSNHnj9/npaTlJTk4uKyePHiNWvWbNq0qa6ujmEYPp9P1/NZtGhRUVHR/v37aRIcHBxcU1Pz0Ta+f/9+yZIlwqeZmZkHDhyov0OTLRW1PTY21tjYmBASEBDw4MGDa9eu0ftf58+fn5OT02Tn0FlASUlJjWNLSUlZtGgR7fnx48f/8ccf4hvr6+vbpUuXoKAgLy+v+fPnh4SE0C5iGKasrMzDw0NNTc3GxubmzZtz58719vY+ceJEg6OTlJRkb29PazQ3N3dxcXF2djY1Ne3SpQshJDIyUszhE4YdEBBAD7T4PoyKitLW1iaEzJ49+/bt2x89UlgKEwAAAECIw9SbPhEXF0eXKhf/98Dx48erq6udnJyKi4uLi4vz8/Ozs7MZhvnxxx/b/LcGSJ6fn9/Bgwffv3/PYgwMw1hZWV25cqVr166SLZnO84mPj5dssQAAAAAdUYun5WRnZwcGBtL7IDU0NOgFaRcXl/3790s+uvZBLww3KSYmpv46lSApFy5cGDdunMQzewAAAACor8XJfVZWVkFBwYYNG9zd3QcNGlReXn79+vXk5OTQ0ND2iK89dLYlWcrLy+kdrhwOR8pVp6amBgQEDB48OCcn5/Lly1KuHQAAAKCzafFqObNmzVq7du327dtHjBiho6MzYcKEkpKSLVu2NPitU5ARO3fuTE5Orq2t9ff3T01NlXLtmpqalZWVt2/fjoyM1NLSknLtAAAAAJ1Na+bcUwKBQFlZWfoXgwHqw5x7AAAAAKHWLIVJ4VI9AAAAAIBMac2PWAEAAAAAgAxCcg8AAAAAICeQ3AMAAAAAyAkk9wAAAAAAcgLJPQAAAACAnEByDwAAAAAgJ5DcAwAAAADICST3AAAAAAByAsk9AAAAAICcQHIPAAAAACAnkNwDAAAAAMgJJPcAAAAAAHKC13iTl5eX9OOAzqCkpERTU5PD4UiwzGvXrtnY2EiwQAAAAICO63+Se0NDQ09PT7ZCAflWUVFx6dKlrl279u7du2/fvurq6hIp1sbGxtbWViJFAQAAAHR0HIZh2I4BOotnz54dOnRo9+7djx49MjMz8/HxmTt3bq9evdiOCwAAAEBOILkHaWMY5sqVK/v27UtISHj//r27u/v8+fPd3d25XC7boQEAAAB0bEjugTUCgeDo0aN79+69ePGivr7+vHnzvvzyy379+rEdFwAAAEBHheQe2Jefnx8bG7tz585nz57Z2tr6+PjMnj1bWVmZ7bgAAAAAOhgk9yArampqzpw5s3v37jNnzvTo0WPevHkLFiwwMjJiOy4AAACADgPJPcicgoKC33//PSoqqqCgwM3NLTAw0M3NTUEBv8kAAAAA8BFI7kFG1dXVXbx4cevWradPn9bX1/f19V28eLGWlhbbcQEAAADILiT3IOvu37+/Y8eOffv2VVVVzZw5c9myZUOHDmU7KAAAAABZhOQeOoaKiorY2NitW7feu3fPyckpKCjIzc1Nsj92CwAAANDRYR4zdAyqqqr+/v65ubmXL19WUlL67LPPTExMtm7dKhAI2A4NAAAAQFbgyj10SHfv3o2IiIiNjVVXVw8MDFy8eLGmpibbQQEAAACwDMk9dGAvX77cvn37jh07Kisr/f39v/76awMDA7aDAgAAAGANknvo8CoqKnbv3r1p06YXL15Mnz79u+++GzRoENtBAQAAALAAc+6hw1NVVV26dOnjx493795969atIUOGeHh4XL9+ne24AAAAAKQNyT3ICUVFRR8fn5ycnCNHjhQWFtrY2Dg7O6ekpLAdFwAAAID0ILkHuaKgoODp6Xnr1q3z58/X1taOGzfuH//4R3JyMttxAQAAAEgDknuQT87OzhcvXkxLS9PQ0HBxcRk1atTJkydxhwkAAADINyT3IM9oTp+ZmWlgYDBp0qThw4fHx8cjxQcAAAB5heQe5J+FhUVcXFxmZqaxsfG0adMsLCyQ4gMAAIBcQnIPnYW5uXlcXNytW7eMjIymTZs2cuTI06dPsx0UAAAAgCQhuYfOZfjw4ceOHcvMzDQ0NPTw8LC1teXz+WwHBQAAACAZSO6hMzI3N09KSsrKyjI0NHR2dra3t7948SLbQQEAAAC0FZJ76Lw+/fTTuLi49PR0ZWVlR0dHe3v7y5cvsx0UAAAAQOtxpHZbYXp6+rNnz6RTF8igqVOnsh2COBcvXlyzZs3Vq1c9PDxCQkLMzc3ZjggAAACgxaSX3Ht5eSUkJEinLpBBHWJ1mrNnz65ZsyYzM3P69OkhISFGRkZsRwQAAADQAlKdluPp6clA53PkyBFpDrO2GD9+/K1bt86dO3f37l1TU9OAgIDCwkK2gwIAAABoLsy5B2jIyckpKysrNjb2/PnzxsbGq1atKi0tZTsoAAAAgI9Dcg/QBAUFBS8vr/v372/evDkmJqZ///7h4eHv379nOy4AAAAAcZDcA4jUpUsXf3//Bw8e+Pv7//Of/zQ1Nd27d29dXR3bcQEAAAA0Dck9wEdoaGiEhoY+fPhw/Pjxfn5+I0aMwKL4AAAAIJuQ3AM0i76+/q5du3JycoyNjR0dHZ2dne/evct2UAAAAAD/A8k9QAuYmJjExcVduHChpKRk+PDhAQEBL1++ZDsoAAAAgP9Acg/QYuPGjcvIyDh06NC5c+cGDBgQHByMe20BAABAFiC5B2gNupxOXl7eDz/8sHnz5oEDB0ZFReFeWwAAAGAXknuA1lNRUVm5cuXff//t7u6+aNEia2vry5cvsx0UAAAAdF5I7gHaSl9fPzIyMjs7W0dHZ8yYMR4eHo8ePWI7KAAAAOiMkNwDSIaZmdnp06eTk5OfPn1qZmYWEBBQXFzMdlAAAADQuSC5B5AkJyenrKys3bt3Hz9+3MTEJDw8vKqqiu2gAAAAoLNAcg8gYQoKCj4+Pg8ePAgICAgODh4yZEhSUhLbQQEAAECngOQeoF2oq6uHhobev3/f0tLy888/d3Fx+fvvv9kOCgAAAOSczCX3SUlJhoaG9+7dYyuAmpqaK1eufP/99+fOnWt1IXw+39fXl8PhcDgcV1fX2NhYCUbYpPj4eBsbG1rj0qVLs7Ky2rtGaI7evXsfOnTo+vXrZWVln3766dKlS9++fct2UAAAACC3ZC65V1VV1dHR6dq1K1sB3Lx58/fff9+4cWN+fn6rC3Fyctq9e7e2tjYhJCYmZtasWZIL8H8UFhbSB15eXhEREYQQCwuLrVu3WlhYtFON0AqWlpbp6em7d+8+fPiwiYkJVsQHAACAdiJzyb2zs3NGRka/fv3YCsDW1varr76SSFHq6uqEkO7du0uktMbevHnj7e0tfKqhodGu1UFb0In49+/fnzp1Kl0R/9q1a2wHBQAAAPJG5pJ7WdClSxeJlMPhcIT/SpxAIJg+ffqTJ0+kUx1IRI8ePbZu3Xrr1i1lZWU7OzsfH5+ioiK2gwIAAAD5IVvJ/Zs3b/bs2ePs7CxcXSQrK2vevHnh4eGTJk1ydnYW//Lc3NzVq1ebmJgUFBSEhIT06dNn8ODBKSkplZWVQUFB/fv37927d/2Z9C9fvvTz8wsJCfHz85syZcqrV6+aLJZhmF27di1cuNDa2trFxeXhw4d0e1pamqGh4dmzZ5vTtKysrBUrVhgZGVVUVPj6+mppaVlZWdHUPC8v7/vvvzczM3v+/PnkyZN79uxpZWVFL+sePnxYXV3d0NCQEFJWVhYSEsLlcm1tbQkhx44du3fvXklJiZ+f3y+//NKcGJps7/Hjx7t168bhcCIiIj58+EAISU9P19PT27hxo6i2FxQUhIWFDRky5PXr166urn369BHVddAkCwuLy5cvJyUl/fXXXyYmJlu3bq2pqWE7KAAAAJALjLR4enp6enqK3ycvLy8oKIgQkpCQQLcMHDgwNTWVYRiBQGBvby/+5UVFRbNnzyaE+Pv7Z2RkvH371tra2sjIKDAwMC8v7927d6NGjTIyMhLu7+DgMG3aNPrY3Nzc29ubPs7JySGE7N69mz4NDQ3du3cvwzA1NTVmZma6uroVFRUMw5w+fVpZWTk2NlZUPAMGDCCElJeXMwxTWFjo5ORECAkMDMzNzc3MzFRSUpo+fTrDMKtWrdLQ0OByuUFBQSkpKYmJiVpaWioqKs+fP2cYxsXFxcDAQFjm0KFDbWxs6OMJEyb07dtX+F90MRYHBwdR8Yhq76pVqwghN2/epE+rqqqsra3FtP3s2bOmpqZcLnfdunVRUVFWVlYFBQVijsuRI0ekOdI6kIqKinXr1ikpKZmamp4/f57tcAAAAKDDk60r94MGDZo0aZLwaXV19cOHDzMyMgghysrKy5cvF/9ybW1tGxsbQsjixYuHDx/erVs3Nze3J0+e+Pr6Dho0SE1NzdHR8cmTJ8LfDeVwOObm5vTxkCFDsrOzG5f5/PnziIgI+jcDl8v19PR88eLFyZMnCSHu7u7v3r2bOXNmc5qmq6traWlJCFm/fr2ZmZmFhYWlpSVtWmhoqLu7u4KCQnh4uIODw+eff75z506BQLBr1y5CiIqKSv1yVFVVm1Ndk0S1NzAwkMfjRUZG0qfJyckTJkwQ03Y3Nzc7O7va2lpvb28/P7/r16/r6+u3OqrOTEVFJTg4OCcnp3///i4uLh4eHk+fPmU7KAAAAOjAeGwH0BCP99+QFBUVXV1dly1blpOTExYWNnny5I++nMvlEkIUFP7zR4uBgQEthz7t3bs3IaSkpISuY3Px4kVCSGVlZWxs7I0bNxiGaVzg1atXq6urAwIChFt8fX2VlZXrV9dMdGdhAw0MDB49ekQfq6iocLlcYZyTJ09WUlK6e/du8wtvDlHtNTAw8PLyOnjwYGhoqJaWVlxc3Lp164jYtisqKvJ4PPrVBLTRgAEDTp06xefzv/rqKzMzsxUrVqxatYrFBaMAAACg45K55L6BxMREPz+/6OjoY8eOxcXFjR07tkUvb3B3KX0qXIWwtrb2p59+unXr1pIlS0StXnLv3j1VVdXo6OjWtqA1eDyevr6+xOdhi2lvUFDQ4cOHo6Kivvnmm5KSEiMjI8JS2zstJyenzMzMTZs2hYaGxsbGRkREfPbZZ2wHBQAAAB2MbE3LaYzH48XGxsbGxvJ4PDc3Nwn+uFVdXZ27u3teXl5iYuKYMWNE7aaiopKfn99gzXvhxJ72IxAITE1NJVXaw4cPBQKBmPZaWlra2dlt37791KlTHh4edCNbbe+0unbt+v3339+/f9/W1tbDwwOzdAAAAKClZDq5r6qqioqKIoTMnDnz2rVrDMOkpKRIqvAbN26cP3/ewcGBPq2urm5yWs7QoUMZhlm5cqVwy+PHj3fs2EEfi/8pIlpgk8WKV1hYWFxc7OnpSQjh8Xjl5eW1tbX0v8rLy4WVKigolJeXN6iuyTAWLFiQmZkpvr3Lly9//vz58uXLvby86BbxbYd28sknn+zfvz85OfnBgweDBw/+5Zdfqqur2Q4KAAAAOgaZS+7pT64KLw/HxMTQvFZfX7979+7Dhw8X//K3b98SQoQTWujTkpIS+vTdu3eEkKqqKvJ/U3T27dt39+7dmJiY3Nzcly9fZmdnv3z5kr6qoqKCEOLs7GxpaXno0KEvvvji4MGDO3bsCAgICAwMJITw+fwePXokJCSID6asrIw+pQ+EsRUVFQkEAuHOVVVVd+7coY83bNgwZ84cKysrQsjQoUNLS0tDQ0MfPHiwYcOGqqqq+/fvZ2Zm0j4pKSnJyMi4dOmSQCCg5ZeWltaPoaysbO7cuT169KBz/ZtsL91z4sSJvXv3Njc319TUpFvEtJ3+vdGgLpAgR0fHnJycNWvWrFmzZsSIEWlpaWxHBAAAAB2B1Nblac5SmBcuXBg9ejQhZOTIkefPn6+srLS0tHR1dQ0LC/P394+Ojv7oyz/99FNCyKxZsx49enTp0qVhw4YRQtzc3LKzs1NTU+nfBt7e3o8fP2YYZsGCBd26dbOxseHz+WfOnNHS0vL09Lx48eL48eMJIcOHDz99+jTDMK9evZo1a5aOjo62traPj49w2ceLFy/q6eklJSU1jiQlJWXRokW0h8ePH//HH3/w+fy+ffsSQhYtWlRUVLR//341NTVCSHBwcE1Nja+vb5cuXYKCgry8vObPnx8SElJXV0eLKisr8/DwUFNTs7GxuXnz5ty5c729vU+cOMEwzJ07dwwMDAYOHBgfH5+UlGRvb09rNDc3d3FxcXZ2NjU1pT/IFRkZKaq9dKVOKiAgID4+vn5Dmmx7VFQUvSN59uzZt2/fFn9QGCyF2TaPHj1yc3PjcDizZ88uKipiOxwAAACQaRym5ZNGWodO9oiPj5dOdR2Ln5/fwYMH379/z2IMDMNYWVlduXJF4uu0xMXF0fX1JVtsp3Ly5MnAwMDy8vJ169Z99dVXwvWgAAAAAOrreCmCtmh0+XlonQsXLowbNw4rMMomDw+Pe/fu+fv7L1++3MHBgf7OGgAAAEADsr4UZmNyuVpLeXk5vcO1wdqdUpCamhoQEDB48OCcnJzLly9LuXZoPlVV1bCwsOnTpy9YsGD48OELFy788ccf6eQuAAAAAKrjXbmXPzt37kxOTq6trfX3909NTZVy7ZqampWVlbdv346MjNTS0pJy7dBSFhYWV69e3b1798GDB01NTRMTE9mOCAAAAGQI5txDu8Oc+/bw4sWLb7/99sCBAxMmTPjtt9/69OnDdkQAAADAPly5B+iQdHV19+/fn5KS8ujRIzMzs+Dg4A8fPrAdFAAAALAMyT1AB+bg4JCZmfnNN9+EhYVZWVnduHGD7YgAAACATUjuATq2rl27rl+/Pjs7u2fPnqNGjQoKCqK/vwYAAACdEJJ7AHkwcODACxcuxMTEHDhwwMTEJCkpie2IAAAAgAVI7gHkBIfD8fHxyc3NHTdu3JQpU6ZOnSqX68YCAACAGEjuAeRKr1699u/ff+rUqevXr5uYmERFRWGdIgAAgCRPQfkAACAASURBVM4DyT2AHPrss8/y8vL8/f0XLVo0duzYhw8fsh0RAAAASAOSewD5RH/R9vLlyyUlJebm5uHh4bW1tWwHBQAAAO0LyT2APBs1alRmZua6devWrVs3cuTIjIwMtiMCAACAdoTkHkDOKSoqrly58u7duz169LCxsVm6dCnWygQAAJBXSO5BSh4/fsx2CJ2asbHxhQsXtm/fvnfvXnNzcz6fz3ZEAAAAIHlI7kFKBgwYoK+vP3Xq1KioqNzcXLbD6Yw4HI6/v39ubq6ZmZmLi4u/v39ZWRnbQQEAAIAkIbkHKbl169bSpUvfvHmzbNmyIUOGINFni4GBwYkTJ/74448TJ04MHjz49OnTbEcEAAAAEsOR2hrYXl5e+fn5QUFB0qkOZEd6enpERIRwpNXU1Ny5c4fP5/P5/NTU1MrKSj09PXt7eycnJ2dn5379+rEbbedRWlq6cuXKqKgoLy+vnTt3ampqsh0RAAAAtJVUk/uEhATp1AUyqMmRhkSfdWfOnAkICKiurt6xY8fnn3/OdjgAAADQJtJL7gHEE5Pou7i49O3bl+0A5VZZWdm3334bHR3t6em5fft2bW1ttiMCAACAVkJyD7Lo/fv3GRkZaWlpfD7/ypUrVVVVwkTf1dW1T58+bAcoh/78809/f/+qqqrffvvNy8uL7XAAAACgNZDcg6wTCAS3b9+un+gbGRnZ2dnZ29sj0Zest2/frlixIjo6+rPPPouMjNTX12c7IgAAAGgZJPfQkYhJ9N3c3Hr37s12gPLg8uXL8+fPf/XqVVhYmL+/P9vhAAAAQAsguYeOSiAQXL16NTU1NS0t7fLlyx8+fECiLykCgeCf//znzz//PH78+MjIyE8++YTtiAAAAKBZkNyDPKioqEhPT2+Q6Ds5OdnZ2Y0dO9bQ0JDtADuk1NTUL7/8sri4ODw8HJfwAQAAOgQk9yBvxCT648aNMzAwYDvAjqSiomLVqlU7duxwd3fHLHwAAADZh+Qe5BlN9Onamjdv3kSi3zp0Fv7r16+3bds2c+ZMtsMBAAAAkZDcQ2dRP9G/ceNGdXW1MNF3dHTEtHLx3r9/v379ejoLPzo6Wk9Pj+2IAAAAoAlI7qEzKi8vv3btWuNE38nJaezYsVpaWmwHKKP4fP78+fMrKyt37tyJn7MFAACQQUjuobMTk+iPGzdOU1OT7QBlC10LPyoqysvLa9euXT179mQ7IgAAAPgvJPcA/yVM9Pl8fmZmZl1dHRL9Jv3555++vr61tbVRUVEeHh5N7vP+/XtlZWUpBwYAANDJIbkHaNq7d++uX7+ORF+U0tLSlStXRkVFzZ49e/v27d26dav/v3fv3vXx8UlLS1NRUWErQgAAgE4IyT3AxzVI9Akhpqam9vb2Tk5Ojo6OnXlqSnx8/KJFi9TU1GJiYsaOHUs3VlVVDRs27N69e/Pnz9+9eze7EQIAAHQqSO4BWgaJfgMvX75csGDB8ePH/fz8Nm/erKqq+u23327ZsqWmpoYQEh8f7+npyXaMAAAAnQWSe4DWKy4uvnbtWlpaGk30ORyOiYkJTfSdnJx69OjRijLfvHmjrq7O5XIlHm27io+PDwgI6Nmz59dff/3VV1/V1dURQjgcjqqqak5OTp8+fdgOEAAAoFNAcg8gGfUT/du3bysoKFhYWNjZ2dFcv/mJ/s8//xwXF7d3797Bgwe3a8ASV1hYOG/evGvXrpWXl9fW1tKNioqKI0aMSE1N7XB/rgAAAHRESO4BJE9Mou/s7KyhoSHmtW5ubufOnePxeMHBwd9++62ioqLUwm47b2/vI0eO0Ak5Qlwud+3atWvXrmUrKgAAgM4DyT1A+yoqKrp+/XozE/2ampru3bsLBAJCCJfLHThw4IEDB0aMGMFS7C1z7NgxUb9sxeFwLl686ODgIN2IAAAAOh0k9wDSU1RU9Ndff6WmpqalpTVI9F1cXLp37379+nUbGxvh/jwer66ubsWKFevXr1dSUmIx8o8qLCwcNGjQ27dvm/xIUVBQ0NXVzc3NFf+tBQAAALQRknsAdhQWFqakpFy6dOmvv/568OCBoqKipaWlnp7eiRMnqqur6+/J5XIHDBhw4MABS0tLtqIVj2EYd3f3P//8k8vlCmfbN6CoqOjh4ZGYmCjl2AAAADoVJPcA7Hv+/DlN9FNTUx88eECXmqlPeAk/ODi4a9eurAQp3r/+9a/k5OTk5OQ///yzvLxcSUmpqqqqwT4cDmf37t1ffvklKxECAAB0BkjuAWRF/Qn3TeJyuQYGBvv37x89erQ0A2uR2trarKwsPp9/9OjRmzdvEkK4XK7wFtuuXbtmZmaampqyGiMAAIDcQnIPICsaTLhvEpfLZRhm2bJlGzZsUFZWlk5grVZcXEyv5Z8+ffr169eKiorV1dVDhgy5deuWjN9CAAAA0FExbXDkyBG2wwc55Onp2ZZhyS78GitAY0eOHGH7rflfbHeGrGP7+PwXcgzxZOpciXMfuxp8xvIkUmLbCwGgtmzZwnYIbWVjYxMUFNSKF27cuDE7O5sQwvxf/qGgoKCmpta9e/eePXv26NFDQ0Oje/fu6urqPXr0UFdX19DQ6NatmyRDl4rKysq8vLxBgwbJ/jcPIBHTpk1jO4SGli1bZmtry3YUMic9PT0iIoLtKBpCjtEkGTxXtvrcB23U+DNWAsn91KlT214IABUfH892CG1lYGDQujfFo0eP5s6dq6Ojo6en16tXLx0dHS0tLYmHByBlMpjc29ra4szVJBlM7nGkmiSD58pWn/ugjdoluQcAiVi9ejXbIQAAAEDHpsB2AAAAAAAAIBlI7gEAAAAA5ASSewAAAAAAOYHkHgAAAABATiC5BwAAAACQE0juAQAAAADkBJJ7AAAAAAA5geQeAAAAAEBOILkHAAAAAJATSO4BAAAAAOQEknsAAAAAADmB5B4AAAAAQE507OS+vLyc7RAAJOPdu3dshwAA/4XzSweCgwWNdeZR0VGT+9jYWCcnJ2NjY7YDaSs+n+/r68vhcDgcjqura2xsbHvXGB8fb2NjQ2tcunRpVlZWe9cI4kVGRo4ZM2bQoEFsB0IIIQzDbNmyJSwszNjYePbs2bW1tWxH1CbZ2dmbN2/+9ddf//3vf4vf8+jRo2PHjqXvi1GjRtnb2w8bNszGxmblypWPHz+WTrSEkPj4eCsrKw6Ho6Sk5OTkNH78eDc3tzFjxvTq1YvD4Tx8+FBqkXRmzTm/lJaW/vDDD999910zy5SRASZ/PnqwDh06NHLkSHV1dWtr6zNnzjSnTBwsmZWUlGRoaHjv3j3xu0kqRezAI4FpgyNHjrSxhJZ6/vw5fVBTU+Pg4KClpSXN2tuPtrY2ISQ/P7/9qhB2HcMw6enphBALC4v2q67VPD09PT092Y6i9VoRf01Njb29va6ubjuF1CLBwcEBAQEMw1y5csXDw+P9+/ctLaH+SGPRkydPvLy8nJycHj161MyX5OfnE0L69Okj3HLjxg03Nzcul7t69era2tp2CbSRtLQ0QoidnV39jdXV1aNHj87Ly2tFgawfEULIkSNH2I2hvo/G89Hzy4kTJ6ZOnUoIWbx4cfPrlZEBJob0z+niNSce8Qdr8+bN48ePj4iIWLp0qYqKCofDSU5Obk7Vsn+wZO1cKZ14zp8/P3z48CdPnojf7aNv4eZ/Ksr+SGCa+kzrSFfu37x54+3tTR9zuVwDAwN245EgdXV1Qkj37t3bqfz6XUcI0dDQaNfq5E9UVFRAQMClS5fq6uokXrhMDeYdO3b07duXEGJvb3/ixImuXbu26OUNRhpbbt26ZW1traend/78+f79+zfzVaqqqoQQZWVl4RZLS8vTp09PmzZt48aN4eHh7RJrIz179iSEKCoq1t/I4/EWLFjA4XBaWpqMHBEp+Pnnn5csWZKenk5PdW3x0bekh4dHdHR0S4uVkQEmC7y8vMLDw58+fdr2osQcrPLy8lOnTp0+fXrp0qURERF8Pp/D4fz888/NKRYHi3r58qWLi8vevXvLysrYjoUQQpydnTMyMvr16yd+N/Fv4RZ9KnbQkdBhknuBQDB9+vQnT56wHUi7oOfsVpy5m6Nx17VrdXLpzZs3UVFRY8eO1dXV/eabbzIyMtiOqF1UVlYWFRW1emDIyJu0pKRkwoQJxsbGmzZtalFbmtxZQUFhx44dOjo6GzZs+OjcHokQFfOMGTNMTU1bVJSMHBHpKC4u3rZt26hRowwNDb///vucnJx2rU5JSamlL5GRASYLbt26tWrVqn79+tnY2OzYsaO4uLg9arl+/XpYWJiw221tbYcNG/bo0aPmvBYHi6qrq0tOTp43b562tvakSZMSEhIqKyvZDqpNWvqp2EFHQrsn97m5uatXrzYxMSkoKAgJCenTp8/gwYNTUlIqKyuDgoL69+/fu3fvc+fO1X9JYmLi4sWLv/nmm/Hjx//www9VVVWEkGPHjt27d6+kpMTPz++XX34R7vzixYvJkyf37NlzxIgRwmlYDMPs2rVr4cKF1tbWLi4udKJqQUFBWFjYkCFDXr9+7erq2qdPn1evXomJ/OXLl35+fiEhIX5+flOmTKm/c0ZGhr+//8yZM62srCIjI2tqasTUSwhJS0szNDQ8e/Zsc3osKytrxYoVRkZGFRUVvr6+WlpaVlZWdCDm5eV9//33ZmZmz58/p622srK6du0aIeTw4cPq6uqGhoaEkLKyspCQEC6Xa2trK6brWtr248ePd+vWjcPhREREfPjwgRCSnp6up6e3ceNGSfW5LKPn8uLi4l9//XXkyJEGBgarVq36+++/W13g8ePH/f39V65c+dVXXxUWFgq3N9mTYkYF/d958+aFh4dPmjTJ2dlZTDli7Nu3z8/PjxASHx/v5+dHL0iIKqTJEdJgpIkZk00ODFF1Ndk6MVatWvXy5cs1a9bweLwG/9Wid6JQ9+7dp06dKhAI4uLiRPVJex+g4OBg8a+S5SMiNfRNWlBQ8MsvvwwdOtTY2Dg4OLiZyVyTmjy/iNFxBxgrGIa5cePGkiVLdHV1bW1to6Ki3r592+rSGh8sR0dHS0vL+vt0796dfjNJcLBaqLq6+syZM9OmTdPQ0PD09Dx58mR1dXVLCzl37hyPx+vSpcupU6cqKyv9/Pw4HI6JicmlS5cIIf/+979tbGw8PT2JiA558+bNnj17nJ2dk5KShGWKyseoxqOicUYknyOhLbN8mjMfrqioaPbs2YQQf3//jIyMt2/fWltbGxkZBQYG5uXlvXv3btSoUUZGRsL9t2zZMmrUqA8fPjAMU1JSYmxsPGbMmLq6OoZhJkyY0LdvX+Ge3t7eqqqqy5Yt+/vvv7Ozs1VVVSdMmED/KzQ0dO/evQzD1NTUmJmZ6erqVlRUnD171tTUlMvlrlu3LioqysrKqqCgQEzkDg4O06ZNo4/Nzc29vb3p46dPn6qqqv7rX/9iGMbHx4cQMmLEiGXLlomql2GY06dPKysrx8bGiqprwIABhJDy8nKGYQoLC52cnAghgYGBubm5mZmZSkpK06dPZxhm1apVGhoaXC43KCgoJSUlMTFRS0tLRUWFzh5zcXExMDAQljl06FAbGxv6uEHX0ZTUwcGhpW1ftWoVIeTmzZv0aVVVlbW1tQT7nJG9eYRUWFhY4wt1NHc0NjZet26dcApgM+OPjY21tramM9qLi4u1tLSEc+6b7Ekxo4JhmIEDB6ampjIMIxAI7O3txZQjPqqSkhJCyIYNG4RbRBUiaoQ0GGmixmSTA0NUXU22TpR3796pqqoqKyuvW7fO0tJSQ0PDycnpzp079H/FvxNLS0sJIaampo3/6+DBg4SQefPmieoTCR6gBm/P2travLw8YVQd7ogw0ppzv2LFisZvUjq76dNPP42IiCgsLGx+PGLOLxS9ftlgzn2HGGBiSG3OvTDDFuJyuQoKCoqKiuPHj9+3bx8NtZnxfPRgUTU1Ndra2jExMfRpRz9Y0jlXPn/+vHHeSM996urqs2fPTk5OphlaM+OZMWNGly5daOs+fPhgaGjo5OQk/F8vL6/Hjx8zIjokLy8vKCiIEJKQkED3F5OPiRkVDT4VO/pIYJr6TJPGDbXbt28nhGRnZ9On69atI4RkZmbSp2vWrCGEFBUVMQzz8uVLVVXV/fv3C1/7+++/E0IOHDjANJXcd+/evbq6mj4dO3asnp4ewzAFBQW9evUS3uWwdu1aQsgff/zBMMz8+fMJIQ8fPmxO68aOHbtx40b6eNasWZ9++il9vGLFCkNDQ/qYnoYjIyPF18swTE1NjZi66if3DMPQFRhKSkroU3t7e2NjY/p45syZioqK9I8fhmHi4+MJIWvXrmUYZvLkyfVP2zY2Nq1O7kW1/dmzZzwez9fXlz49depUSEiI+La3qM+ZDpXcUxwOR1FRkcPhWFlZRUREeHh4fDT+iooKPT29Q4cOCbdMmTKFJvdielLUqPjw4QOHw9m6dSvdfuzYMfHliNEguRdTiKgR0mCkiRmTDQaGqLqabJ0Yly9fJoTY29vTC8+PHj0yMTFRU1Orfy++qNeK+RCn3y46OjpK4QDRt6e6ujrtrpEjR2pra2toaIh/lcweEYbV5F74JuVyuRwOx8bGJjIyspnJfZPnF6Emk3umIwwwMVhM7oXox6mKioq3t/e3337bnHg+erCoxMREZ2dnmolSHfpgsZjcC3Xp0oUQ0qtXryVLljg4ODQnnosXLxJChJn0smXLunTp8vr1a4Zh3r9///nnnzNiO4Re4xcm96LyMUbsqGjwqch08JHANPUZ2/Br6/bA5XIJIQoK/5kCRO9yEN4u1rt3b9pybW3ta9euVVRU0C3UhAkTCCEpKSlN3v2gqKgo/ObdyMiILgJz9erV6urqgIAA4W6+vr70Zgi6P82kP4oOwcrKytjY2Bs3bjD/d5NWQUGBQCCgj01MTDQ1NZ89eya+XmEnNBPdWdg0AwMD4TfLKioqXC5X2HuTJ09WUlK6e/du8wtvDlFtNzAw8PLyOnjwYGhoqJaWVlxcHP1TTVJ9TiUlJcna/QAcDod+ijXGMAz9dvLmzZs3btxQUFDQ09MrKCj45JNPRJV25cqVwsLCoUOHCrcIkxIxPSlqVCgqKrq6ui5btiwnJycsLGzy5Mniy2k+MYWIGiHN12BgiKqrydaJQU9FM2bMoLel9u/f/6effpo0adKOHTtCQkJIC9+JQvRmsoEDB0rtAA0fPjwlJYU+rq6upl/adsQjQk2bNm3atGktDalFxNz8zTAMXdT1+vXrdB7j9u3bHR0dNTU1xRTY5PnlozrKABNDCh+/YpZzoB+nAoGAXhYlhPz8889ff/21+I796MF68+bNhg0bzp49W791Hf1g3bp1i91zJZ2g+/Lly19//ZUQoq6ufurUKZq2ieLg4NCvX78DBw7MnDmTEJKdnV1TUxMfH+/v75+YmPjFF18QsR3SYLKlqHyMav5buKOPhMakkdw30GAs0qd0ERJ67/zr16+F/yucdtL8Yu/du6eqqtqKpQwaqK2t/emnn27durVkyRJra2t6SiCEuLu7Hzp06MKFC46OjqWlpRUVFW5ubhKst0V4PJ6+vn6DSWZtJ6rthJCgoKDDhw9HRUV98803JSUlRkZGRNJtt7S0pF+9yY7jx48nJCSI+l8Oh6OgoMAwzD/+8Q+BQPDJJ5+IyewJIfQCQ5N/LbSuJxMTE/38/KKjo48dOxYXFzd27FiJHBExhYgZIRKvq3HrxJSjo6ND/veT2sHBgRCSl5fXxvAIIebm5qwcIEVFRXoJsyMeESooKIhO7m8/hw8fFrOKOX2TEkKcnJzOnTs3f/588Zl945dLIETR2B1gDdA5xO1q2bJlYlZf4fF4tbW1ampqVlZWFy5cWL58ufDiYHM0ebCCgoIiIiJ69erVmnD/l+wcLHrxoqWvapE3b97UzzUbU1RUrK6uNjAw6Natm6GhofjMnhDC4XDmzJkTEhLy4sWLR48eWVlZcbncgwcP0uSe/tRP8ztEVD7WZL0fLa2lZGckNMZCci8GXd6o8V3MLVojQkVFJT8/Pz8/v/5CSMXFxXQt+Waqq6tzd3fX0dFJTEwkhOzevVv4X7NmzXr//r2Pj8+XX35ZUFBw+PBhOzs7SdXbCgKBoKVraIjx8OHDTz75ZMqUKU22nRBiaWlpZ2e3fft2U1NTDw8PulGybf/kk0+8vLza0AjJe/LkSZPJPY/Hq6mpMTY2/vLLL318fPT09JoTOU3rnz59OnDgwAb/1bqe5PF4sbGxn3322fLly93c3LKysiRyREQVoqmpKerd0WpiAm7cOjE/+EXfC/VvUFZXV1dUVOzRo0erY2MYJiEhQVFR0c3NLSEhgZUD5O7uTjrmEaFsbGza+0198+bNxhs5HA59k1paWs6cOXP69On058BausZru5KFAVafFD5+6R+rDXC5XIZhuFyus7Pz3LlzJ02alJSUdOHChRZl9k3avn375MmTR48e3cZyiIwdrB49erT3wSosLGwyuadvq+7du0+bNm327Nl2dnb0lx+aY86cOevXrz98+PCNGzc2bdqUkpIye/bsixcv6unp0avUze8QUfmYFMjUSGhMtpbCtLW1VVdXr38fdH5+vkAgmDhxIiFEQUGhOT8mPHToUIZhVq5cKdzy+PHjHTt2tCiSGzdunD9/nl7zI4TQaVvCxw8fPrxz505ISEhMTIzwW2nx9YpfH50W3opv0gsLC4uLi+nd5Twer7y8XPiTouXl5cJKG3SdqIoYhlmwYEFmZqaotlPLly9//vz58uXLhR8rEunzDoRm5wMGDPj+++/v379///79lStX6unpNfPln376KSGETm+l6urq6IFrRU9WVVVFRUURQmbOnHnt2jWGYVJSUlp3RBocaFGFiHl3NBhpYsZkA6LqarJ1Ypqgp6fn4ODA5/OFW0pKSqqrq21sbOhTMe9EUe+LTZs23b17d+XKlX369JHCARLzadARjwhb6NzF/v37r169+tGjR9evX1+6dKlELtyKJ/sDTNbQOyIUFBRGjx79+++/v3r16vTp015eXqJmQrbUoUOHlJWV688fE34+4GC1FL0jQlVVdfr06SdOnCgpKYmMjLS3t2/RdfG+ffs6ODhs27ZNWVlZX19/ypQpampqs2bNmjdvHt2h+R0iKh8Tr3EyKX8jQRpX7uniVsKpI/QpvXWPEPLu3TtCCF3vUlNTMzw8fNGiRfRLFkLIr7/+OmfOHPq1r76+fklJSUZGxrt376ysrF69elVaWvrhwwf6EVBUVFRVVSUQCJydnS0tLQ8dOlRZWTllypS3b98ePXr0jz/+IITQs1ppaSn9FScx6Ejdt2+flZXVzZs3c3NzX758mZ2d3atXr+jo6L/++svCwkJPT09NTU1TU5N+4SCmXj6f/8UXX+zZs4dm4aK6qKysTE1NjfzfRC5hjxUVFQlnldG+unPnjrm5OSFkw4YNc+bMsbKyIoQMHTo0ISEhNDR06tSpcXFxVVVVz549y8zMHDZsWIOuo+XT20SEysrKlixZ0qNHDzo5rMm201PjxIkTe/fubW5uLvxeWyJ9LuOqq6vp94+6urpz5syZMWMGPQStYGdnN3bs2L17944YMWLOnDm5ubmpqanFxcWHDx+eOHGiqJ4UMypiYmIWLlzI5XL19fW7d+8+fPhwa2trUeWIQX+KT1isqMNKf3a7yRHSYKSJGZMNBoaYIdS4deJb8dNPP9nb2585c4Ze7Y6NjTU3N587dy752DuRftzXf689ffp006ZNv/3229KlS9evXy8+TkkdIPrGbPJCRgc9IlIjfJP26dOHvklb/a2mqPOLiooK3aGiooIQIvwzieoQA0x20Ok3o0aN8vHx8fT0pPfJtIKYg3XmzJlt27bNnTuX3kjNMEx2draZmZmTkxMOVvPRL094PN7EiRO9vb3d3Nxa8TsP9c2bN8/Hx+fo0aOEEBUVFS8vrxs3bowcOZL+r5iOpd/KCn8YITw8vMl8jIgdFQ0+Fa9evSqHI+GjN+GK0Zw76y9cuECvU86aNevRo0eXLl0aNmwYIcTNzS07Ozs1NZWeGLy9ven6RwzDJCUlubi4LF68eM2aNZs2bRLe237nzh0DA4OBAwfGx8fv37+ffs++dOnSsrKymJgY+rmwdOnSqqqqV69ezZo1S0dHR1tb28fHhy6/GBUVRb/amD179u3btz/augULFnTr1s3GxobP5585c0ZLS8vT07O8vPzkyZPdunWr34eDBw+mVTRZL8Mw9PumpKSkxrWkpKQsWrSIljN+/Pg//viDz+fTlQQWLVpUVFS0f/9+mvEHBwfX1NT4+vp26dIlKCjIy8tr/vz5ISEhwv4pKyvz8PBQU1OzsbG5efPm3Llzvb29T5w40aDrkpKS7O3taY3m5uYuLi7Ozs6mpqb0DUDvNBfVdmHYAQEB8fHx9RsikT5nZHi1nJ49ewYGBqamptZfbKGxZsZfVlY2b968Xr169e7dOzg42N/ff968eXw+v7a2tsmeFDMqKioqLC0tXV1dw8LC/P39o6OjaRWiRqMoGRkZM2bMIIT069cvNja2tLRUTCGiRkj9kcaIHpNNDowm66qsrGyydeLdunXLw8Nj4cKF69atW7JkSVlZGd0u5p2YlJQknDtub2/v6Ojo7u4+fvz4r7/+WriSppg4JXWAkpKS6OQBDofz3Xff5ebmNoizIx4RIq3Vcnr16hUUFETvJ25LPOLPLwzDnD9/nq7xYGRkFBkZKVyLSfYHmHhSWy3H2NjY3Nx806ZN+fn5bYxHzMFKTU1tfDOikpISXUqrox8sqa2Ww+PxXF1dDxw48PbtW0nF8/79+yVLlgifZmZm0kURhZrskAsXLtCPx5EjR54/f55hGFH5mPi3cINPxY4+EpimPtM4TBt+qTsuLo6urNzqEjqi48ePV1dXOzk5FRcXFxcX2udDgQAAIABJREFU5+fn01U+f/zxRynU7ufnd/Dgwffv30uhLlEYhrGysrpy5Up7zFulU33oEp+y4+XLl5qamo1/FKkx2YwfgEUcDufIkSPNn5LbOi9evNDR0WnO/GzpxNMRSe2c/vz5c319fdmJpyOSzrmmqqrq3bt3WlpaMhJPA+zmY7Kj8WeabN1QK2Vi7lGIiYkR3i1aX3Z2dmBgIJ29oKGhYWxsTAhxcXHZv39/+8Upay5cuDBu3DiZuiOtvUlhqq50tGLMyyD5aAVIlq6uLtshQHM1J7MHWaCkpNTGGTjtB/mYGJ06uRdO22q+rKysgoKCDRs2uLu7Dxo0qLy8/Pr168nJyaGhoe0RYWPl5eX0bjnpr26bmpoaEBAwePDgnJwc+lNB0OG0YszLIPloBQAAtBrr+Zgsk63VcmTfrFmz1q5du3379hEjRujo6EyYMKGkpGTLli3Cu6za1c6dO5OTk2tra/39/VNTU6VQY32ampqVlZW3b9+OjIxszpd0AAAAAO2B3XxMxnXqK/etwOVy169fv379eoFAoKysLOXL5wsXLly4cKE0a6xv0KBBdFEOAAAAABaxm4/JOCT3rYQ/DQEAAADYhXysMUzLAQAAAACQE0juAQAAAADkBJJ7AAAAAAA5geQeAAAAAEBOILkHAAAAAJATSO4BAAAAAOQEknsAAAAAADmB5B4AAAAAQE4guQcAAAAAkBNI7gEAAAAA5ASP7QAA4D8yMzN5PJ62traOjo6CAv7wBgAAgBZDcg8gK37//fdt27YRQhQUFDQ0NLS1tQ0MDPT19XV0dPT19bW1tXV1dXV1dXV0dLS1tTt09i8QCFRUVNiOAgAAQA5JILnncDhtLwRAyNPTk+0Q2iQhIaGNb4q6urrXr1+/fv36/v37kooKAISmTZs2bdo0tqOAZkGOIYqsnSvbfu4DSeEwDNPqF+fn51+9elWC0QAQQgwNDW1tbdmOopXS09OfPXvWute+e/fO19f3o7txOBw1NbWAgABLS8vWVSQ1Hz58+Pvvv+/evZuZmZmfn08/bYKCgmxsbNgODaRq1KhRBgYGbEfxH3FxcWyHINOmTp3Kdgj/gRxDPJk6V7bl3Adt1+Aztk3JPQBIlqmpqZir9QoKCnV1dV988UVkZKSmpqY0A2u+urq6zMxMPp9/7ty5tLS0Dx8+dO3atbKykhCioKDg5+e3a9cutmMEAACQW5hzDyATKioq0tLS1NXVu3Tp8uHDh8Y7KCoqdu/ePTo6evLkydIP76MKCwtTU1OTk5OTkpKKi4u7dOlSU1NTV1dHCKGZPY/H69+//5YtW9iOFAAAQJ4huQdgjUAguHr16qVLly5dunTjxo3q6mo9Pb3q6uoGu9EL9hMnToyKiurZsycbkX7EqlWrwsPDuVwuIaS2tpYQ0vjvEx6Pd+zYMWVlZRbiAwAA6DQwLQdAqt6/f5+RkZGWlsbn869cuVJVVaWnp2dvb+/k5OTq6tqtWzctLa3670pFRUVtbe29e/c6OzuzGLZ4r1+/HjRo0KtXr2hm3xiHw9mzZ8+8efOkHBgAAEBng+QeoN3VT+hTU1MrKyuFCb2Li0vfvn3r72xmZnbv3j1CCJfLraur8/X13bRpU7du3dgJvdmSk5NdXV2b/DxRVFScNGlSfHy89KMCAADobJDcA7SLmpqaO3fu8Pn8xgm9s7Nzv379RL1w2bJlv/32G4fD0dfX37dvn4ODgxSjbhN/f/+9e/c2mFbE5XINDAyys7PV1dXZCgwAAKDzwJx7AIkRldBv3bpVfEJfn4ODw7Zt2wIDAzdu3KiqqtreMUsKwzAWFhaqqqrl5eU1NTXC7RwO5+jRo8jsAQAApANX7gHapH5Cn5aW9v79e+EVeicnJyMjo5YW+ObNm7y8PDs7u/aItp08e/bMz8+Pz+dPnz798OHDdJEcQoiCgsKWLVuWLFnCbngAAACdB5J7gBYTk9Db2dkNHjyY7QClKj4+fsGCBT169IiJiRk9evTy5ct//fXXmpoaRUVFR0fHM2fO4DcLAQAApAbJPUCzCBP61NTUy5cvv337VldX9x//+EfnTOipwsLCBQsWnDp1ytfXd/PmzXQSUWVl5dChQx8/ftyrV6/c3FzZXLsTAABAXiG5BxAJCb0YDS7Y1/+vW7du2dvb8/l8e3t7tsIDAADonJDcA/yP2trarKwsmtBfuXKlrKysV69eo0ePtrOzs7e3Hz58OCaZFBYWBgQEnD59uv4F+wZycnKGDBki/dgAAAA6OST3AP9J6FNTU9PS0s6fP4+EXoz4+PiAgICePXvu2bNnzJgxbIcDAAAA/wPJPXRS9RP65OTk0tJSHR2dMWPGIKEXpTkX7AEAAIBdSO6hE2kyobeysqIL3SChFwMX7AEAADoEJPcg5+on9Hw+/82bN9ra2tbW1kjom+np06cBAQF8Pn/ZsmUhISHKyspsRwQAAAAiIbkHOVRbW/v333/TbB4JfavV1dXt2LHju+++MzAw2LNnz6hRo9iOCAAAAD4CyT3Iibq6unv37tGE/sKFC69fv9bS0rKxsaEJ/bBhwxQUFNiOsSN5/Pixn5/flStXli9fHhwc3LVrV7YjAgAAgI9Dcg8dWOOEvlu3btbW1k5OTkjoW62mpmb79u2rV68eMGDAnj17Ro4cyXZEAAAA0FxI7qHjefLkCZ1vg4Re4rKzs+fPn5+Tk7Ny5crVq1d36dKF7YgAAACgBXhsBwDQLMKE/uLFi69evaIJ/bfffouEXlKqq6s3b968du3akSNH3r59e9CgQWxHBAAAAC2GK/cguxok9GpqajY2NrhC3x7S09Pnz5//9OnTtWvXrlixAn0LAADQQSG5B9kiTOhTUlJKSkqECb2dnZ21tbWioiLbAcobgUDwz3/+85dffnFycoqMjOzTpw/bEQEAAEDrtXJaTnp6+ubNmyUbCnRyz549y8rKqqqqUlRU1NLSsrW1XbdunYWFBZfLZTs0ufXXX3/5+fkVFxfv2LHDz88PK4QCAAB0dK388v3Zs2cJCQmSDQU6OXV1dRMTE0dHx4kTJ3K5XCUlpREjRiCzbyelpaUBAQFjx441MTHJycnx9/dHZg8AACAH2nRDbXx8vKTiAKjPy8uL7RDk2ZEjR5YtW8bhcBITE6dMmcJ2OAAAACAxuG0OoBP5f//v/3322WczZsyYMGFCbm4uMnsAAAA5g+QeoFOoqanZunXr0KFDHzx4kJycHB0d3aNHD7aDAgAAAAlDcg8g/7KyskaNGrVixYrAwMCcnBxHR0e2IwIAAIB2geQeQJ4JBIJVq1aNHDmya9euWVlZYWFhSkpKbAcFAAAA7QW/UAsgt06fPh0YGFhWVrZp06avvvoKP00FAAAg93CyB5BDL1688PHxmTBhgpWV1f3795cuXYrMHgAAoDPAlXsAucIwzIEDB77++mt1dfWzZ8+6ubmxHREAAABIDy7mAciPBw8eODo6zp8/f9asWdnZ2cjsAQAAOhsk9wDyoLKyMjg4+NNPPy0tLU1PT9+6dauamhrbQQEAAIC0YVoOQId37tz/b+9Og5rK0j6AnxgQBcGFRUFQGzdaRNRuAri8ogIuAy5taOkG1C4DCKgMg446dDdOoaKOPa3WiMKIKA2KLNOIoKMGsCQ2NAgIsqioNZYssgoakLCd98OdSjFCECUhIf5/H6x7b07O85wTPzy5nHtyY/v27dXV1X/729+8vb3ZbLa8MwIAAAD5wJ17gCGsvLz866+/Xrly5Zw5c0pKSnbs2IHKHgAA4FOG4h5gSGJ+cdbU1DQ3NzclJSUhIcHQ0FDeSQEAAICcobgHGHoEAsH8+fN3797t5eVVXFy8evVqeWcEAAAACmGoFvdCoVDeKQDIQUNDg6+v75IlS3R1dQsLCw8fPjxixAh5JwUAAACKYugV99HR0ba2ttOnT5d3IlLG5/N5PB6LxWKxWCtWrIiOjpZ1xLi4OCsrKyair6/v/fv3ZR0RBoJSGhkZOXPmzPj4+IiIiNTUVBMTE3knBQAAAIqFRSn9iLfFxsZu3Ljx4977caqqqvT19QkhnZ2dtra2RUVFtbW1gxZ90Ojp6dXW1paXl0+cOFFGIcQzSQjJysqytraeO3dufn6+jMJ9HCcnJ0JIXFycvBNRFPn5+d7e3vfu3fP29j5w4ICmpqa8MwIAAABFNDTu3L969crV1ZU5ZrPZSvzgoJaWFiFk9OjRMuq/+0wSQsaMGSPTcDBwjY2Nvr6+FhYWqqqqeXl5J06cQGUPAAAAkgyBfe5bWlqcnZ2fPXsm70QGA4vFEv8rdT1nUqbhYODi4uJ27NjR3t4eEhLi7u6OTwoAAAD6JsM798XFxX/5y19mzpxZUVERFBQ0efJkU1PT9PT01tZWPz+/qVOnTpo06caNG93fkpCQsH379l27dq1ater7778XiUSEkF9//bW0tLSurs7d3f3YsWPixi9fvly3bt24ceO++OKL0tJS5iKl9MyZM15eXpaWlvb29mVlZYSQioqKw4cPz549u6GhYcWKFZMnT66vr5eUdq+Nq6ur3d3dg4KC3N3d169f3/3t165d8/b29vX1tba2/uc//9lHGoSQu3fvGhkZXb9+vT8TeP/+/d27dxsbGzc3N/N4PB0dHQ6Hw5TmJSUlAQEBs2bNqqysZCaBw+FkZWURQi5duqSlpWVkZEQIaWpqCgoKYrPZ1tbWfcxkH3od+JUrVzQ1NVks1vHjx9va2gghmZmZ+vr6hw4dkspHAISQx48f29vbOzs729vbP3r0yMPDA5U9AAAAvB/9KJcvX37ve2tqatzc3AghHh4eubm5r1+/trS0NDY29vHxKSkpefPmzYIFC4yNjcXtf/755wULFrS1tVFK6+rqpk+fvmTJkq6uLkqpg4PDlClTxC1dXV01NDT++Mc/Pnz4sLCwUENDw8HBgXkpODj4/PnzlNKOjo5Zs2ZNmDChubn5+vXrJiYmbDY7MDAwLCyMw+FUVFRISrvXxjY2NswzBpRSc3NzV1dX5jgyMtLZ2bmzs5NSevDgQUJIamqqpDQopSkpKSNHjoyOjpYUfdq0aYQQoVBIKa2qqrK1tSWE+Pj4FBcX5+fnq6mpOTs7U0r37t07ZswYNpvt5+eXnp6ekJCgo6Ojrq5eWVlJKbW3tzc0NBT3aWZmZmVlxRy/M5MPHz4khNjY2EjKR9LA9+7dSwjJyclhTkUikaWlpbQ+Akopl8vlcrl9NFBiTU1Nu3btUlVV/fLLL8UzDAAAANAfMizuKaWnTp0ihBQWFjKngYGBhJD8/Hzm9IcffiCE1NTUUEqrq6s1NDQiIyPF742IiCCE/PLLL7S34n706NHt7e3M6dKlS/X19SmlFRUV48ePZ0ptSumPP/5ICImJiaGUbt26lRBSVlbWn9H1bLx06dJDhw4xxy4uLnPmzKGU1tTUjB49+tmzZ8z12trar776qqSkpI80KKUdHR19hO5e3FNK9+3bRwipq6tjThctWjR9+nTm+Ntvv1VVVWW+C1FKmWdPf/zxR0rpunXruhf3VlZWH13c9zpwSumLFy9UVFR4PB5zmpycHBQURKX3EXyaxX1XV1dERMSECRPGjRsXEhIinkYAAACAfpLtmns2m00IGTbsv4t/mAdhVVVVmdNJkyYxlauurm5WVlZzczNzheHg4EAISU9P7/4AqJiqqqqKyn+TNzY2zszMJIT89ttv7e3tnp6e4mY8Hm/kyJHi9kzp/F49G6elpRFCWltbo6Ojs7OzKaWEEIFA0NXV9dlnnzFtdHR0EhISCCHx8fGS0hDPST8xjcUjNTQ0fPLkCXOsrq7OZrPFk7lu3To1NbUHDx70v/P+6HXgTCZOTk5RUVHBwcE6OjqxsbHMNzdpfQSfoNzc3J07d2ZlZbm4uPz000+6urryzggAAACGnkF9oPadRcPMaVdXFyHk+fPnhJCGhgbxq+J1Jv3vtrS0VENDQ7zwXYo6OzuPHj167969nTt3WlpaMqvbi4qKmL8evDMu2aXRBxUVFQMDg46ODul22+vAGX5+fpcuXQoLC9u1a1ddXZ2xsTGR09iHupcvXwYGBp49e3bx4sV5eXnm5ubyzggAAACGKkXZCpO5/91zS5wP+pkedXX18vLy8vLy7hcHvh1+V1fX6tWrS0pKEhISlixZIr6upaXV2tpaUlLSvbFIJJJRGu/V0tIixV81Kisra2lp6XXgDAsLi4ULF546dSo5OdnR0ZG5KK+xD1Ht7e0nTpyYOXNmSkpKREREeno6KnsAAAAYCEUp7q2trbW0tBITE8VXysvLW1pa1qxZQwgZNmyYUCh8bydmZmaU0j179oivPH36NCQkZIC5ZWdn37x508bGhjll7tYTQiwsLAgh33//PfPHB0JIbm5uSkpK32mIG/eK6Zl++K+DVVVV1dbWcrlcQoiKiopQKOzs7GReEgqF4qDvzKSkQJTSbdu25efn9zpwMX9//8rKSn9/f+Y3p4jMPgKlxOfz586du2/fPi8vr4cPH27atAn74QAAAMAAyXZZzuvXrwkh4rUizGldXR1z+ubNG0IIs9+ltrb2kSNHvL29U1NTly9fTgg5efLk5s2bly5dSggxMDCoq6vLzc198+YNh8Opr69vbGxsa2sbPnw4IaSmpkYkErW0tNjZ2VlYWFy8eLG1tXX9+vWvX7/+17/+FRMTQwhhit3GxkbmZ5v69k5jpuS6cOECh8PJyckpLi6urq4uLCycOnXqqlWrEhMTly9fzuVynz9/3tDQcPbsWUqppDT4fP6GDRvCw8OZKlzSjDU1NY0aNYo56D6BNTU1LS0t4sYikaigoIC513vgwIHNmzdzOBxCiJmZWXx8fHBw8Ndffx0bGysSiV68eJGfnz9v3rx3ZpLpv7GxsXsOTU1NO3fuHDt2LLPWv+fAx48fP378eELImjVrJk2aZG5urq2tzbxXWh+BcisrK/vTn/6UnJzs4OCQnJwsfmwDAAAAYKA+7jnc/uyWk5qaOmfOHEKIi4vLkydPbt++PW/ePELIypUrCwsLBQLB/PnzCSGurq5Pnz5l3pKYmGhvb799+/Yffvjhp59+YvbBpJQWFBQYGhrOmDEjLi4uMjJy7NixhBBfX9+mpqZz586NGzeOORWJRPX19S4uLnp6erq6ups2bWL2WwwLC2MeT3Rzc8vLy+s77V4bb9u2TVNT08rKis/nX7t2TUdHh8vlCoXC5uZmLy+viRMnjh8/3svLq7GxkWnfaxqU0rS0NH19/cTExJ5x09PTvb29mQ9l1apVMTExfD5/ypQphBBvb++amprIyEim4t+/f39HRwePxxs+fLifn5+Tk9PWrVuDgoLE09XU1OTo6Dhq1CgrK6ucnJwtW7a4uromJSW9M5OJiYmLFi1iIpqbm9vb29vZ2ZmYmDBfmUJDQ/sYuDhtT0/PuLi47gMZ+EdAlXe3HKFQGBgYqKamZmJi8u9//1ve6QAAAICyYdEPXwFCCImNjWW2Px/I9woYCHd396ioqLdv38oxB0oph8PJyMgYMWKEdHtm1vkw+3sqB0rpL7/8smfPHpFIFBgY6OPjI94ECQAAAEBaPtHyoo99Bs+dOyd+PBT6lpqaumzZMqlX9soH21wCAADA4PhEi3sl2L9FKBT2uhHnIBAIBJ6enqampkVFRXfu3Bnk6ENLZWVlQEDAhQsXlixZkp+fzyxUAwAAAJARRdktBz7I6dOnb9261dnZ6eHhIRAIBjm6trZ2a2trXl5eaGiojo7OIEcfKpjl9TNmzLh9+/bly5fT09NR2QMAAICsfaJ37oc6Ly8vLy8veUX//PPPnz59Kq/oiq+rqysqKmrv3r1v3rzZtWvX3r17sXIJAAAABgfu3ANIE5/Pnz9/Po/Hc3R0fPr06f79+1HZAwAAwKBBcQ8gHSUlJQ4ODnZ2drq6uvn5+aGhoXp6evJOCgAAAD4tKO4BBqqystLT03POnDnV1dW3b9++deuWqampvJMCAACATxHW3AN8vObm5n/84x8HDx4cM2ZMSEgIj8cbNgxfmAEAAEBuUNwDfIzuT836+/vjqVkAAABQBLjLCPDB+Hz+vHnz8NQsAAAAKBoU9wAfQPzUrJ6eHp6aBQAAAEWD4h6gX54/f/7dd9+ZmZnV1dVlZGTgqVkAAABQQCjuAd7j5cuXO3bsmDFjRkZGRnR0dGZm5qJFi+SdFAAAAEAv8EAtgEQNDQ0nT578+9//rqWldfTo0W3btqmpqck7KQAAAACJUNwD9ILZ4/Lw4cNsNjsgIGDnzp0jR46Ud1IAAAAA74HiHuB/tLW1nT9/PjAwUCgU+vj47Nu3b/To0fJOCgAAAKBfUNwD/Fd7e/ulS5f2799fXV3N4/ECAgKwEw4AAAAMLQMq7p2cnKSVB0B3WVlZVlZWgxauq6srISEhICDgP//5z3fffRcYGGhgYDBo0QEAAACk5SOLeyMjIy6XK91UAMSsrKysra0HJxafz//zn/9cUFCwYcOG69evT506dXDiAgAAAEgdi1Iq7xwA5EMgEAQEBGRkZPzhD384ePDgnDlz5J0RAAAAwIBgn3v4FGVmZi5fvnzx4sUaGhr37t27evUqKnsAAABQAiju4dNy9+7dFStWLFiwoL29PSMj49q1a/Pnz5d3UgAAAADSgeIePhV37951dHRctGhRc3NzUlLSnTt38EOzAAAAoGRQ3IPyEwgETFn/6tWrpKQk5lTeSQEAAABIH4p7UGYCgcDW1nbx4sWvXr3i8/ko6wEAAEC5obgHJUQpvXr16oIFCxYvXsxmswUCgUAgWL58ubzzAgAAAJAtFPegVDo7Oy9evGhubr527Vptbe3MzMwbN24sXLhQ3nkBAAAADAYU96Ak2traIiMjTU1N3dzcJk+enJ2dffXq1cH8mVsAAAAAufvIX6gFUBxCoTA8PPzYsWM1NTUbN268cuXKzJkz5Z0UAAAAgByguIchrLq6+tSpU6dOnWpra/Pw8PD39zcwMJB3UgAAAAByw6KUyjsHgA9WVFT0888/R0dHa2lpeXt779ixQ1tbW95JAQAAAMgZinsYYgQCwZEjR1JSUqZOnbp9+3Z3d3d1dXV5JwUAAACgEPBALQwNIpEoPDzczMxs8eLFb9++TU5Ofvz4sa+vLyp7AAAAADGsuQdFV1lZGRYWFhIS0tTUtHbt2vDwcA6HI++kAAAAABQRluWAgqKU3rp1KyQkJDk5efz48R4eHh4eHvr6+vLOCwAAAEBx4c49KJzXr1/HxMScPHmyuLj4iy++OHfu3DfffKOqqirvvAAAAAAUHYp7UCDZ2dlnz569ePEii8Vyc3OLiYmZPXu2vJMCAAAAGDKwLAfkr7GxMSoq6uzZswUFBbNnz/b09Ny0aZOWlpa88wIAAAAYYlDcgzzl5uaGhYVFR0d3dnY6Ojp6eHjY2trKOykAAACAoQrFPchBVVVVVFRUeHj4o0ePLCwseDzeN998o6mpKe+8AAAAAIY2rLmHwdPa2pqUlHThwoUbN25oaWm5uLhcvnzZ3Nxc3nkBAAAAKAncuYfBkJubGxkZefHixYaGhmXLlrm5uXG5XPz+FAAAAIB0DUZxn5mZ+eLFC1lHAcWUnJyclpZWUVFhaGhoY2Nz9OhR7FUPAAAAICODUdw7OTnFx8fLOgoMCfhLEQAAAIDsDBucMFwul8Kn7fLly4Pznw0AAADgkzVIxT0AAAAAAMgainsAAAAAACWB4h4AAAAAQEmguAcAAAAAUBIo7gEAAAAAlASKewAAAAAAJYHiHgAAAABASaC4BwAAAABQEijuAQAAAACUBIp7AAAAAAAlgeIeAAAAAEBJoLgHAAAAAFASKO4BAAAAAJQEinsAAAAAACUxBIr7jo6OjIyMgICAGzduyK7PxMREIyOj0tJSaYUghPD5fB6Px2KxWCzWihUroqOjpdh5r+Li4qysrJiIvr6+9+/fl3VEAAAAAFAcKvJO4P1ycnIiIiIiIiKMjY1l16eGhoaent6IESPEbaqqqvT19QcSxdbW1tbWNikpqba29ty5cxMnThxQ0pKJU3VycjIyMrK2tp47d+6JEydkFA4AAAAAFNMQuHNvbW29Y8cOWfdpZ2eXm5v72WefMaevXr1ydXWVSiwtLS1CyOjRo6XSW0/vpDpmzBiZhgMAAAAAhTUEintCyPDhwwezz5aWFmdn52fPnkklEIvFEv8rdT1TlWk4AAAAAFBkilLcl5SUBAQEzJo1q7Kyct26dePGjeNwOFlZWT1bXrp0SUtLy8jIiBDS1NQUFBTEZrOtra0JIRUVFYcPH549e3ZDQ8OKFSsmT55cX19fXV3t7u4eFBTk7u6+fv36+vr6nn2+evUqPDzczs4uMTGREPLrr7+WlpbW1dW5u7sfO3bsypUrmpqaLBbr+PHjbW1thJDMzEx9ff1Dhw4RQu7evWtkZHT9+vX+DPP+/fu7d+82NjZubm7m8Xg6OjocDocpzfuYgT6G/E6q/cmh1wnpe4yU0jNnznh5eVlaWtrb25eVlUma7f4kAAAAAACyQmWPy+Vyudy+2+zdu3fMmDFsNtvPzy89PT0hIUFHR0ddXb2yspJSWlRURAg5e/Ys09je3t7Q0FD8XjMzMysrK0rp9evXTUxM2Gx2YGBgWFgYh8OpqKiwsbHZuHEj09Lc3NzV1ZU57t5nSUmJn58fISQ+Pp551cHBYcqUKd3TI4Tk5OQwpyKRyNLSkjlOSUkZOXJkdHS0pKFNmzaNECIUCimlVVVVtra2hBAfH5/i4uL8/Hw1NTVnZ+f3zoCkIfdM9eHDh4QQGxsbSflImpA+xhgcHHz+/HlKaUdHx6xZsyZMmNDc3NzrbEsKSim9fPny4Px/AwAAAPhkKcqd++Dg4NWrVw8bNuzIkSM2NjZfffXV6dOnW1pazpw507Oxurp691MNDQ3mYOXKlQsXLuzs7HR1dXXDMh0yAAAF3ElEQVR3d//9998NDAxYLJa5uTnTYPbs2YWFhT07/Pzzz9euXdtHej4+PioqKqGhoczprVu3HBwcmOPVq1e/efPm22+/7c8wJ0yYYGFhQQj561//OmvWrLlz51pYWOTm5r53BiQN+SNImhBJY6ysrDx+/LibmxshhM1mc7ncly9fXr16tdfZ/uisAAAAAGDgFGi3HHV1dTabraqqypyuW7dOTU3twYMHH9SJqqqqiooKc7OckZaWRghpbW2Njo7Ozs6mlPb6RhWVvqbC0NDQyckpKioqODhYR0cnNjY2MDBQ/Cqbze5/hkxjcThDQ8MnT54wx1KZgfeSNCGSxvjbb7+1t7d7enqKe+DxeCNHjiS9zTYAAAAAyJGi3LnvSUVFxcDAoKOjY4D9dHZ2BgcHu7i4TJs2zdLS8qP78fPza21tDQsLa2trq6urk+K+nJJIawbe0ceE9DrG0tJSDQ2Nf/6vNWvWSDcrAAAAABg4xS3uCSEtLS0mJiYD6aGrq2v16tUlJSUJCQlLliwZSFcWFhYLFy48depUcnKyo6PjQLrqv4HPQHdlZWUtLS19TEivY1RXVy8vLy8vL+/esra2VlpZAQAAAIC0KG5xX1VVVVtby+Vye76koqIiFAo7OzuZU6FQ2NXV1Wsn2dnZN2/etLGxYU7b29slLct5x7Bhw4RC4TsX/f39Kysr/f39nZycul+XFJ3BROxn3O66z0AfQ34nVUmBKKXbtm3Lz8/ve0J6jtHMzIxSumfPHnGbp0+fhoSEfOhwAAAAAEDWFKu4F4lEBQUFzPGBAwc2b97M4XAIIa9fvyaENDc3My+ZmZk1NjYGBwc/fvz4wIEDIpHo0aNH+fn5hBCmAm5sbGRaMtu9X7hw4cGDB+fOnSsuLq6uri4sLKyurn6nz6qqKtLthrSBgUFdXV1ubu7t27dbWlqYi2vWrJk0aZK5ubm2trY4Zz6fP3bs2Pj4eEmDYgI1NTUxp8yBeLFNTU2NuP8+ZqCPIb+TKtO/eAbEQbds2TJ27FhmrX+vEyJpjHZ2dhYWFhcvXtywYUNUVFRISIinp6ePj0/P2QYAAAAAORuEHXn6sxUmpZTH4w0fPtzPz8/JyWnr1q1BQUFdXV2U0t9//33VqlWEkPnz56ekpFBKm5qaHB0dR40aZWVllZOTs2XLFldX16SkpLCwMF1dXUKIm5tbXl4e0+22bds0NTWtrKz4fP61a9d0dHS4XG5aWlr3PlNTU//v//6PEPLll1/evHmTUlpQUGBoaDhjxoy4uLjuSXp6er5zJS0tTV9fPzExseeI0tPTvb29mXletWpVTEwMn8+fMmUKIcTb27umpiYyMnLUqFGEkP3793d0dEiagT6G/E6qiYmJixYtYiKam5vb29vb2dmZmJgwv9gVGhoqaUKYnToljbG+vt7FxUVPT09XV3fTpk3Mlpe9znYfsBUmAAAAgKyx6IcvF/lQzAKPuLi4vpu5u7tHRUW9fftW1vl8NEoph8PJyMgYMWKELPpXhBmQ3RhjY2OZ/fWl2y0AAAAAiCnWshwFl5qaumzZMhlV9griUxgjAAAAgLJSoH3uhUIh83wns1BecQgEAk9PT1NT06Kiojt37sgukBxnYNDGCAAAAACyoyh37k+fPn3r1q3Ozk4PDw+BQCDvdP6HtrZ2a2trXl5eaGiojo6OjKLIdwYGZ4wAAAAAIFMKtOYelBvW3AMAAADImqLcuQcAAAAAgAFCcQ8AAAAAoCRQ3AMAAAAAKAkU9wAAAAAASgLFPQAAAACAkkBxDwAAAACgJFDcAwAAAAAoCRT3AAAAAABKAsU9AAAAAICSQHEPAAAAAKAkUNwDAAAAACgJFPcAAAAAAEoCxT0AAAAAgJJQGZww5eXlsbGxgxMLFFNmZqa8UwAAAABQcoNU3GdlZW3cuHFwYgEAAAAAfJpYlFJ55wAAAAAAAFKANfcAAAAAAEoCxT0AAAAAgJJAcQ8AAAAAoCRQ3AMAAAAAKIn/B5fHS1zDy5DTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.utils.plot_model(\n",
    "    model=model, to_file=\"dnn_model.png\", show_shapes=False, rankdir=\"LR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and evaluate our model\n",
    "We've built our Keras model using our inputs from our CSV files and the architecture we designed. Let's now run our model by training our model parameters and periodically running an evaluation to track how well we are doing on outside data as training goes on. We'll need to load both our train and eval datasets and send those to our model through the fit method. Make sure you have the right pattern, batch size, and mode when loading the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 31 steps, validate for 1 steps\n",
      "Epoch 1/5\n",
      "31/31 [==============================] - 2s 61ms/step - loss: 42.7320 - rmse: 5.0342 - mse: 42.7320 - val_loss: 3.9244 - val_rmse: 1.9810 - val_mse: 3.9244\n",
      "Epoch 2/5\n",
      "31/31 [==============================] - 0s 13ms/step - loss: 1.7095 - rmse: 1.2776 - mse: 1.7095 - val_loss: 1.2681 - val_rmse: 1.1261 - val_mse: 1.2681\n",
      "Epoch 3/5\n",
      "31/31 [==============================] - 0s 12ms/step - loss: 1.2811 - rmse: 1.1199 - mse: 1.2811 - val_loss: 1.1632 - val_rmse: 1.0785 - val_mse: 1.1632\n",
      "Epoch 4/5\n",
      "31/31 [==============================] - 0s 13ms/step - loss: 1.1373 - rmse: 1.0564 - mse: 1.1373 - val_loss: 1.1155 - val_rmse: 1.0562 - val_mse: 1.1155\n",
      "Epoch 5/5\n",
      "31/31 [==============================] - 0s 13ms/step - loss: 1.1427 - rmse: 1.0564 - mse: 1.1427 - val_loss: 1.0589 - val_rmse: 1.0290 - val_mse: 1.0589\n",
      "CPU times: user 5.21 s, sys: 844 ms, total: 6.05 s\n",
      "Wall time: 3.71 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "tf.random.set_seed(33)\n",
    "\n",
    "TRAIN_BATCH_SIZE = 32\n",
    "NUM_TRAIN_EXAMPLES = 1000 * 5  # training dataset repeats, it'll wrap around\n",
    "NUM_EVALS = 5  # how many times to evaluate\n",
    "# Enough to get a reasonable sample, but not so much that it slows down\n",
    "NUM_EVAL_EXAMPLES = 1000\n",
    "\n",
    "trainds = load_dataset(\n",
    "    pattern=\"./data/babyweight_train*\",\n",
    "    batch_size=TRAIN_BATCH_SIZE,\n",
    "    mode=tf.estimator.ModeKeys.TRAIN)\n",
    "\n",
    "evalds = load_dataset(\n",
    "    pattern=\"./data/babyweight_eval*\",\n",
    "    batch_size=1000,\n",
    "    mode=tf.estimator.ModeKeys.EVAL).take(count=NUM_EVAL_EXAMPLES // 1000)\n",
    "\n",
    "steps_per_epoch = NUM_TRAIN_EXAMPLES // (TRAIN_BATCH_SIZE * NUM_EVALS)\n",
    "\n",
    "logdir = os.path.join(\n",
    "    \"logs\", datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\"))\n",
    "tensorboard_callback = tf.keras.callbacks.TensorBoard(\n",
    "    log_dir=logdir, histogram_freq=1)\n",
    "\n",
    "history = model.fit(\n",
    "    trainds,\n",
    "    validation_data=evalds,\n",
    "    epochs=NUM_EVALS,\n",
    "    steps_per_epoch=steps_per_epoch,\n",
    "    callbacks=[tensorboard_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Need for regularization\n",
    "\n",
    "Let's use a high-cardinality feature cross to illustrate the point. In this model, we are predicting taxifare in New York city using a feature cross of lat and lon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!bq show mlpatterns || bq mk mlpatterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "CREATE OR REPLACE TABLE mlpatterns.taxi_data AS\n",
    "\n",
    "SELECT\n",
    "  (tolls_amount + fare_amount) AS fare_amount,\n",
    "  pickup_datetime,\n",
    "  pickup_longitude AS pickuplon,\n",
    "  pickup_latitude AS pickuplat,\n",
    "  dropoff_longitude AS dropofflon,\n",
    "  dropoff_latitude AS dropofflat,\n",
    "  passenger_count*1.0 AS passengers\n",
    "FROM `nyc-tlc.yellow.trips`\n",
    "# The full dataset has 1+ Billion rows, let's take only 1 out of 1,000 (or 1 Million total)\n",
    "WHERE ABS(MOD(FARM_FINGERPRINT(CAST(pickup_datetime AS STRING)), 1000)) = 1\n",
    "AND\n",
    "  trip_distance > 0\n",
    "  AND fare_amount >= 2.5\n",
    "  AND pickup_longitude > -78\n",
    "  AND pickup_longitude < -70\n",
    "  AND dropoff_longitude > -78\n",
    "  AND dropoff_longitude < -70\n",
    "  AND pickup_latitude > 37\n",
    "  AND pickup_latitude < 45\n",
    "  AND dropoff_latitude > 37\n",
    "  AND dropoff_latitude < 45\n",
    "  AND passenger_count > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "CREATE OR REPLACE MODEL mlpatterns.taxi_noreg\n",
    "TRANSFORM(\n",
    "  fare_amount\n",
    "  , ML.FEATURE_CROSS(STRUCT(CAST(EXTRACT(DAYOFWEEK FROM pickup_datetime) AS STRING) AS dayofweek,\n",
    "                            CAST(EXTRACT(HOUR FROM pickup_datetime) AS STRING) AS hourofday), 2) AS day_hr\n",
    "  , CONCAT(\n",
    "     ML.BUCKETIZE(pickuplon, GENERATE_ARRAY(-78, -70, 0.01)),\n",
    "     ML.BUCKETIZE(pickuplat, GENERATE_ARRAY(37, 45, 0.01)),\n",
    "     ML.BUCKETIZE(dropofflon, GENERATE_ARRAY(-78, -70, 0.01)),\n",
    "     ML.BUCKETIZE(dropofflat, GENERATE_ARRAY(37, 45, 0.01))\n",
    "  ) AS pickup_and_dropoff\n",
    ")\n",
    "OPTIONS(input_label_cols=['fare_amount'], model_type='linear_reg') \n",
    "AS\n",
    "\n",
    "SELECT * FROM mlpatterns.taxi_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bigquery\n",
    "SELECT SQRT(mean_squared_error) AS rmse FROM ML.EVALUATE(MODEL mlpatterns.taxi_noreg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "CREATE OR REPLACE MODEL mlpatterns.taxi_l2reg\n",
    "TRANSFORM(\n",
    "  fare_amount\n",
    " , ML.FEATURE_CROSS(STRUCT(CAST(EXTRACT(DAYOFWEEK FROM pickup_datetime) AS STRING) AS dayofweek,\n",
    "                            CAST(EXTRACT(HOUR FROM pickup_datetime) AS STRING) AS hourofday), 2) AS day_hr\n",
    "  , CONCAT(\n",
    "     ML.BUCKETIZE(pickuplon, GENERATE_ARRAY(-78, -70, 0.01)),\n",
    "     ML.BUCKETIZE(pickuplat, GENERATE_ARRAY(37, 45, 0.01)),\n",
    "     ML.BUCKETIZE(dropofflon, GENERATE_ARRAY(-78, -70, 0.01)),\n",
    "     ML.BUCKETIZE(dropofflat, GENERATE_ARRAY(37, 45, 0.01))\n",
    "  ) AS pickup_and_dropoff\n",
    ")\n",
    "OPTIONS(input_label_cols=['fare_amount'], model_type='linear_reg', l2_reg=0.5) \n",
    "AS\n",
    "\n",
    "SELECT * FROM mlpatterns.taxi_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bigquery\n",
    "SELECT SQRT(mean_squared_error) AS rmse FROM ML.EVALUATE(MODEL mlpatterns.taxi_l2reg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.2812030944146013"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "100 * (4.814606 - 4.828183)/4.828183"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2020 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
